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Section  1 
Introduction 

101.  Purpose.  This  guide  is  designed  to  help  you,  the  OTD  (Opera¬ 
tional  Test  Director) ,  perform  your  functions  in  OT&E  (operational 
test  and  evaluation) .  It  presupposes  you  have  attended  the  basic 
course  for  OTDs,  and  that  you  have  familiarized  yourself  with  the 
basic  source  material  listed  in  the  next  paragraph. 

102.  Basic  Source  Material.  It  is  essential  that  you  be  familiar 
with  the  following  in  their  latest  editions: 

a.  OPNAVINST  5440.47,  Mission  and  Functions  of  OPTEVFOR. 

b.  COMOPTEVFORINST  5400.1,  COMOPTEVFOR  Staff  Manual.  Among 
other  things,  discusses  OPTEVFOR' s  mission  and  organization,  and 
identifies  staff  agencies  who  can  help  you. 

c.  SECNAVINST  5000.1  (w/DOD  Directive  5000.1  and  DOD  Instruction 
5000.2),  System  Acquisition  in  the  Department  of  the  Navy.  Promul¬ 
gates  basic  DOD  policies  for  weapon  system  acquisition,  including 
the  formats  specified  for  the  MENS  (Mission  Element  Need  Statement) , 
DCP  (Decision  Coordinating  Paper) ,  and  IPS  (Integrated  Program  Sum¬ 
mary)  and  the  organization  and  functions  of  the  DSARC  (Defense 
Systems  Acquisition  Review  Council) . 

d.  OPNAVINST  5000.42,  Weapons  Systems  Selection  and  Planning. 
Amplifies  SECNAVINST  5000.1  and  specifies  the  format  and  content  of 
ORs  (Operational  Requirements) . 

e.  OPNAVINST  3960.10  (w/DOD  Directive  5000.3),  Test  and 
Evaluation.  The  fundamental  document  in  U.S.  Navy  OT&E. 

f.  OPNAVINST  4720.9,  Approval  of  Systems  and  Equipment  for 
Service  Use.  Gives  the  criteria  for  ASU  (approval  for  service  use) 
and  PASU  (provisional  approval  for  service  use) ,  and  discusses  the 
relationship  between  ASU  and  the  production  decision. 

g.  OPNAVINST  5000.46,  Decision  Coordinating  Papers  (DCPs) , 
Program  Memoranda  (PMs)  and  Navy  Decision  Coordinating  Papers 
(NDCPs) ,  preparation  and  processing  of.  Discusses  each  of  these 
documents  (format  and  content)  in  detail. 

h.  OPNAVINST  5401.6,  Navy-Wide  Tactical  Development  and 
Evaluation  Program.  Established  the  Navy's  TAC  D&E  (tactical 
development  and  evaluation)  program.  Specifies  COMOPTEVFOR ' s 
involvement . 
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i.  NAVSO  P-2457,  RDT&E  Management  Guide.  A  good  orientation 
in  the  Navy's  system  for  managing  RDT&E.  Provides  general  informa¬ 
tion  and  leads  to  directives  containing  detailed  guidance.  Explains 
the  many  standard  forms  associated  with  RDT&E  (e.g.,  DD  Form  1498, 

DD  Form  1634) . 

103.  Additional  References.  The  Headquarters'  Tech  Library 
maintains  classified  and  unclassified  documents  in  a  variety  of 
technical  fields  and  on  a  variety  of  weapons  systems.  Some  of 
the  more  general  references  maintained  by  the  library  which  are 
particularly  useful  to  OTDs  are  listed  below.  The  listing  is  without 
order;  additional  titles  will  be  added  as  recommended  by  you,  the 
OTD . 


a.  Jane1 s.  An  unofficial  but  thorough  series,  well  illustrated 
and  quantified.  (Note:  If  a  number  is  critical  to  your  evaluation, 
check  what  Jane's  says  with  Intelligence.)  The  Jane's  series  includes: 

(1)  Fighting  Ships. 

(2)  All  the  World's  Aircraft. 

(3)  Weapon  Systems. 


(5)  Surface  Skimmers. 

b.  NAVMATINST  3960.6,  Test  and  Evaluation.  Implements 
OPNAVINST  3960.10  within  NAVMAT,  and  establishes  procedures  for 
ACAT  IV  programs.  In  your  dealings  with  the  DA  (Developing  Agency) , 
your  knowledge  of  his  own  rules  may  help  you. 

c.  DICNAVAB  and  CAAL  (COMOPTEVFOR  Acronym  and  Abbreviation 
List) .  Unofficial  but  useful  lists  of  acronyms. 

d.  ASTM  E  380-74,  Metric  Practice  Guide.  Describes  the  metric 
system  directed  by  DOD  Directive  4120.18.  Provides  conversion 
factors  for  most  non-metric  units  of  measure. 

e.  Roget's  Thesaurus  or  Webster's  New  Dictionary  of  Synonyms. 
Can  help  you  avoid  using the  same  word  over  and  over. 
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f.  OPNAVINST  5200.6,  Procedures  for  Maintaining  Navy 
Backup  Files  and  Preparation  of  Point  Papers.  Gives  the 
standard  Navy  format  for  point  papers,  and  hints  for  pre¬ 
paring  them. 

g.  JCS  Pub  1,  Dictionary  of  Military  and  Associated 
Terms.  Provides  standardized  (and  sometimes  surprising) 
definitions  for  use  throughout  DOD. 

h.  O'Hayre,  John,  Gobbledyqook  Has  Gotta  Go.  Useful 
writing  tips,  with  emphasis  on  writing  in  the  Federal  gov¬ 
ernment. 

i.  DOD  Telephone  Directory.  Who's  where  in  the  Wash¬ 
ington  DOD  complex,  organizational  structure,  etc.  Handy 
for  figuring  out  "copy  to's." 

j.  Payne,  Stanley  L. ,  The  Art  of  Asking  Questions. 

Particularly  useful  to  OTDs  who  have  to  develop  question¬ 
naires  -  helps  avoid  "loaded"  questions,  etc. 

k.  Ship  Acquisition  Reef  Points.  A  useful  supplement 
to  NAVSO  P-2457  for  OTDs  involved  with  ship  acquisition 
programs . 

l.  Project  Analysis  Methodology.  The  text  for  02B's 
course  in  analysis.  Covers  what  the  OTD  generally  needs  to 
know  about  analysis,  and  discusses  the  support  provided  by 
Division  Analysts. 

m.  M.G.  Natrella,  Experimental  Statistics.  An  excel¬ 
lent  cookbook  for  the  OTD  stuck  without  an  analyst. 

n.  S.  Siegel,  Nonparametric  Statistics  for  the  Behav¬ 
ioral  Sciences.  An  excellent  cookbook  for  analysis  of  count 
type  data  (hits/misses,  yes/no).  Each  technique  is  illus¬ 
trated  with  a  worked-out  example. 

o.  Naval  Institute's  Fundamentals  of  Naval  Operations 
Analysis .  Not  a  cookbook,  but  an  interesting  discussion  of 
the  application  of  operations  analysis  methods  to  naval 
operations  problems. 

p.  SEMCIP's  The  Commanding  Officer's  Guide  to  the  Ship¬ 
board  Electromagnetic  Environment.  Good  summary  ot  ship¬ 
board  EMI  (electromagnetic  interference)  problems. 

q.  FPMR  101-11.2,  Plain  Letters.  Tips  on  writing 
clearly  and  briefly.  Includes  an  excellent  "Watchlist"  of 
overworked  and  incorrectly  used  words  and  phrases. 


McGraw-Hill  Encyclopedia  of  Science  and  Technolo 
These  15  volumes  cover  many  topics  and  can  save  you  time  in 
researching. 

s.  WPC  (Word  Processing  Center)  Procedures  Manual. 

Tells  you  how  to  get  the  most  out  of  the  WPC  -  including 

instructions  for  two  types  of  dictation  inputs  to  the  WPC. 
These  include  portable  recorders  available  on  a  loan  basis 
from  the  WPC  for  use  on  trips  and  at  meetings. 

t.  Reliability  Computer.  Not  a  document,  but  a  circu¬ 
lar  slide  rule  for  calculating  reliability  of  continuously 
operated  and  one-shot  devices.  This  gadget  is  available 
from  Code  02B  ( free ) . 

u .  Duane ,  J . T . ,  Learning  Curve  Approach  to  Reliability 
Monitoring.  "Duane  Growth  Curves"  are  being  mentioned  more 
and  more  frequently  in  NAVMAT  test  documents.  This  is 
Duane's  IEEE  paper  on  the  subject. 
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DT&E  and  OT&E 

201.  Introduction.  T&E  (test  and  evaluation)  comes  in 
three  varieties:  DT&E  (development  T&E),  OT&E  (operational 
T&E),  and  PAT&E  (production  acceptance  T&E).  Each  variety 
is  discussed  in  detail  in  the  fundamental  Navy  T&E  document, 
OPNAVINST  3960.10.  PAT&E,  testing  on  production  items  to 
demonstrate  that  requirements  of  the  production  contract 
have  been  met,  is  largely  independent  of  DT&E  and  OT&E,  and 
simply  determines  if  the  Navy  got  what  it  signed  a  contract 
for.  DT&E  and  OT&E,  on  the  other  hand,  are  closely  related 
because  they  both  influence  what  the  Navy  thinks  it  wants, 
and  how  it  goes  about  specifying  this  in  a  contract.  This 
relationship  is  discussed  below. 

202.  Definitions 


a.  According  to  DOD  Directive  5000.3,  DT&E  is  conducted 
to  assist  the  engineering  design  and  development  process  and 
verify  attainment  of  technical  performance  specifications 
and  objectives.  DT&E  is  planned  and  conducted  by  the  DA 
(usually  a  SYSCOM). 

b.  According  to  DOD  Directive  5000.3,  OT&E  is  conducted  (r 
to  estimate  a  system's  operational  effectiveness  and  opera¬ 
tional  suitability,  identify  needed  modifications,  and  pro¬ 
vide  information  on  organization,  personnel  requirements, 
doctrine,  and  tactics.  In  the  Navy,  OT&E  is  planned  and 
conducted  by  OPTEVFOR. 

203.  Apparent  Overlap  of  DT&E  and  OT&E.  It  is  a  fact  that 
DT&E  and  OT&E  necessarily  examine  the  same  features  of  a 
system  —  performance  features.  Why  is  it  necessary  that 
DT&E  and  OT&E  both  look  at  the  same  features  of  a  system? 

Because  their  viewpoints  are  completely  different.  This 
fundamental  difference  (viewpoint)  means  that  DT&E  and  OT&E 
are  completely  different;  there  is  no  overlap  or  duplication 
between  the  two.  (If  there  is,  T&E  is  not  being  planned 
properly.)  DT&E  and  OT&E  differ  in: 

a.  The  way  tests  are  conducted. 

b.  What  is  being  tested. 

c.  The  evaluation  criteria. 

d.  The  test  measurements  and  the  data  base. 
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204.  How  Are  Tests  Conducted? 

a.  DT&E,  whose  viewpoint  is  technical,  is  properly  con¬ 
ducted  : 

(1)  In  a  controlled  environment  that  minimizes  the 
chance  that  unknown  (or  unmeasured)  variables  will  affect 
system  performance. 

(2)  By  technical  personnel  skilled  at  "tweaking"  to 
maximize  performance. 

b.  OT&E  is  properly  conducted: 

R)  (1)  In  an  operationally  realistic  environment;  e.g., 

high  seas,  temperature  extremes,  high-density  electromagne¬ 
tic  environments ,  etc . 

(2)  With  fleet- type  operators  and  maintenance  per¬ 
sonnel  . 


(3)  Against  a  simulated  enemy  who  fights  back. 

205.  What  Is  Being  Tested? 

a.  DT&E  tests  a  weapon,  or  a  "black  box,"  whatever  the 
development  program  involves.  (Seldom  does  a  development 
program  involve  a  complete  weapon  system. ) 

b.  OT&E  tests  total  weapons  systems.  If  a  missile  is 
being  developed,  OT&E  does  not  test  the  missile  itself,  but 
rather  the  missile  system,  which  includes  the  firing  plat¬ 
form,  that  platform's  acquisition  system,  the  targeting  sys¬ 
tems,  the  people  who  man  it,  logistics  support,  interfacing 
equipment,  and  so  forth.  Thus,  the  missile  under  develop¬ 
ment  is  hazarded  by  the  fact  that  it  may  fail  OT&E  through 
no  fault  of  its  own,  but  because  interfacing  systems  aren't 
well  enough  adapted  to  it,  etc.  But  there's  no  point  in 
deploying  a  "good"  missile  that  can't  work  in  the  fleet. 

206.  What  Are  The  Evaluation  Criteria?  DT&E  employs  tech¬ 
nical  criteria  for  evaluating  system  performance.  These 
criteria  are  usually  parameters  that  can  be  measured  during 
controlled  DT&E  tests,  that  are  important  to  the  DA  (and  to 
the  contractor,  who  gets  paid  to  meet  them).  They  are  usu¬ 
ally  of  little  direct  use  in  OT&E.  Signal  strength,  rate  of 
climb,  mean  time  between  failures,  etc.  are  important  in 
determining  how  well  the  sytem  was  engineered,  but  frequent¬ 
ly  are  of  no  interest  (per  se)  to  OT&E,  which  is  structured 
to  demonstrate  target  acquisition  at  useful  ranges,  super¬ 
iority  in  air  combat,  and  the  probability  of  accomplishing  a 
mission.  So  DT&E  and  OT&E  criteria  are  different. 
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What's  Measured  And  How  Often? 


a.  In  DT&E,  the  DA  generally  knows  what  he  wants  to 
measure  (e.g.,  some  particular  parameter:  launch  velocity, 
the  number  of  G's  pulled  as  the  missile  acquires,  time-to- 
climb,  etc.).  DT&E  tests  are  structured  to  hold  many  things 
constant,  isolate  others,  and  allow  measurement  of  the  one 
or  two  quantities  of  interest.  In  OT&E,  it  often  is  not 
possible  to  specify  measurements.  The  objective  is  often 
simply  to  create  combat  conditions  as  closely  as  possible 
and  watch  what  happens . 

b.  In  DT&E  it  is  generally  possible  to  verify  data  sta¬ 
tistically  through  replications  of  tests.  In  OT&E  this  is 
often  not  possible,  because  interactions  during  testing  are 
as  unique  as  a  combat  experience  is  unique. 

208.  Combined  DT&E  and  OT&E.  DOD  Directive  5000.3  requires 
that  planning  for  DT&E  and  OT&E  be  coordinated  at  the  test 
design  stages  so  that  each  test  phase  uses  resources  effi¬ 
ciently  to  yield  the  data  necessary  to  satisfy  common  needs 
of  the  DA  and  the  OT&E  agency.  With  regard  to  combined  test¬ 
ing,  DOD  Directive  5000.3  states  that: 

a.  Development  and  operational  tests  may  be  combined 
when  clearly  identified  and  significant  cost  and  time  bene¬ 
fits  will  result,  provided  that  the  necessary  resources, 
test  conditions,  and  test  data  required  by  both  the  DA  and 
the  OT&E  agency  can  be  obtained. 

b.  Participation  by  the  OT&E  agency  in  the  planning  and 
execution  of  tests  must  be  sufficient  to  ensure  that  the 
testing  conducted  and  data  collected  are  sufficient  and  cre¬ 
dible  to  meet  the  OT&E  agency's  requirements. 

c.  When  a  combined  testing  program  is  chosen,  it  will 
normally  include  dedicated  operational  test  events,  and  the 
final  period  of  testing  prior  to  the  Milestone  III  decision 
will  emphasize  appropriate  separate  operational  testing  by 
the  OT&E  agency. 

d.  In  all  cases,  the  OT&E  agency  shall  provide  a  sep¬ 
arate  and  independent  evaluation  of  the  test  results. 
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Section  3 


Concept  of  OT&E 

301.  Characteristics  of  OT&E.  In  general,  OT&E  is 

a.  Planned  by  COMOPTEVFOR. 

b.  Conducted  by  fleet- type  personnel. 

c.  Carried  out  in  a  realistically  simulated  combat 
environment. 

d.  Reported  by  COMOPTEVFOR. 

The  two  distinct  parts  of  OT&E  (i.e.,  testing  and  evaluation) 
are  discussed  below.  As  will  be  obvious  to  the  thoughtful 
reader,  OT&E  is  more  an  art  than  a  science,  and  differs  con¬ 
siderably  from  the  more  widely  understood  DT&E  (See  Section 
2). 

302 .  Operational  Testing 

a.  Operational  testing  consists  of  the  following  ele¬ 
ments  : 


(1)  Exercising  a  system  or  equipment  under  condi¬ 
tions  which  simulate,  as  closely  as  is  possible,  the  expec¬ 
ted  operational,  combat  environment,  and 

(2)  Recording  sufficient  data  during  the  exercise  to 
document  all  operationally  significant  system  or  equipment 
characteristics . 

b.  The  first  element  of  operational  testing  means  that 
the  test  article  is  exercised: 

(1)  In  operational  scenarios  in  which  both  forces 
(ours  and  theirs)  employ  realistic  tactics. 

(2)  Against  targets  that  fight  back. 

Furthermore,  the  test  article  itself: 

(1)  Is  representative  (insofar  as  possible,  consider¬ 
ing  the  stage  of  development)  of  the  intended  production 
equipment. 

(2)  Is  installed  (insofar  as  possible)  as  it  is 
expected  to  be  installed  in  the  fleet. 


(3)  Is  operated  and  (usually)  maintained  by  fleet- 
type  personnel. 

c.  A  few  words  of  explanation.  Operation  by  fleet- type 
personnel  is  always  required  for  operational  testing. 
Regardless  of  the  OT&E  phase,  contractor  operation  voids 
operational  testing.  The  same  is  not  true  of  maintenance . 
During  early  I OT&E  (OT-O/I),  maintenance  by  fleet-tyye  per- 
sonnel  is  usually  not  possible.  Only  the  maintainability 
portion  of  operational  testing  is  voided  by  contactor  main¬ 
tenance.  (Note  that  even  when  there  is  no  operational  test¬ 
ing,  an  operational  evaluation  of  technical  data  is  always 
possible. ) 

d.  The  second  element  of  operational  testing,  the  data 
element,  is  just  as  important  as  the  first.  Unless  the 
correct  data  are  recorded  accurately  during  the  exercise, 
some  (at  least)  of  the  effort  was  wasted. 

e.  A  final  word  on  operational  testing.  Operational 
testing  seeks  to  provide  data  on  system  performance  (where 
performance  includes  all  the  elements  of  operational  effec¬ 
tiveness  and  operational  suitability)  in  the  operational 
environment.  This  environment  includes  many  things.  Among 
these  are  the  people  (operators,  maintainers,  etc.);  the 
other  systems  which  will  also  be  consuming  power,  radiating, 
etc.,  in  the  same  ship  or  aircraft;  ships  or  aircraft  in  the 
vicinity,  employing  their  own  systems;  established  con¬ 
straints  or  rules  of  engagement;  weather  factors  (visibil¬ 
ity,  sea  state,  etc.);  the  simulated  enemy,  and  the  tactics, 
countermeasures,  etc.  he  employs;  and  so  on.  This  large  num¬ 
ber  of  variables,  and  the  fact  that  their  effects  may  change 
as  a  function  of  their  combinations  with  other  variables 
dictates  that  each  operation  or  run  include  as  many  elements 
of  the  whole  as  is  possible.  Technically  oriented  tests 
with  highly  restricted  objectives  (e.g.,  point-to-point 
navigation  runs  which  include  nothing  else)  are  wasting 
operational  test  resources  (particularly,  they  are  wasting 
scarce  fleet  services).  The  way  to  avoid  this  waste  is  to 
structure  the  tests  around  mission-oriented  scenarios  —  and 
do  the  whole  thing  in  an  exercise.  Investigate  point-to- 
point  navigation  as  a  part  of  the  ASW  aircraft's  mission  to 
locate  and  destroy  submarines.  If  the  system  will  be  eoqploye 
in  the  fleet  in  a  variety  of  scenarios  —  investigate  all  of 
them  before  repeating  any.  This  will  insure  the  most  complet 
data  coverage  if  unforeseen  circumstances  cut  testing  short. 
Always  strive  to  maximize  test  variables  while  acquiring  data 
in  areas  not  yet  explorecT  And  because  not  all  variables 
are  identifiable  be!  :ore  testing,  be  alert  for  the  unexpected, 
and  ready  to  record  its  results. 


303. 


Operational  Evaluation 

a.  Operational  evaluation  is  the  analysis  and  interpre¬ 
tation  of  data  from  an  operational  viewpoint,  for  the  pur¬ 
pose  of  predicting  the  operational  effectiveness  and  opera¬ 
tional  suitability  of  a  system. 

b;  Of  a  System!  There  is  really  no  such  thing  as  oper¬ 
ational  effectiveness  (or  operational  suitability)  of  a 
component  or  a  black  box.  Consider  a  new  fuze  in  an  old 
bomb.  If  the  A-6  aircraft  consistently  drops  bombs  which 
fuze  properly  and  destroy  the  target,  the  new  fuze  is  part 
of  an  operationally  effective  weapons  system.  If,  on  the 
other  hand,  the  bombs  consistently  fail  to  damage  the  target 
(for  whatever  reason),  the  new  fuze  is  not  part  of  an  opera¬ 
tionally  effective  weapons  system.  (For  more  on  this,  see 
paragraph  205.)  It  is  crucial  that  the  OTD  understand 
this  —  lest  he  end  up  asking  the  wrong  questions  about  the 
thing  he's  evaluating . 

c.  A  proper  operational  evaluation  proceeds  as  follows: 

(1)  The  objectives  identify  all  the  proper  elements 
of  operational  effectiveness  and  operational  suitability. 

(If  they  don't,  you're  probably  sunk  because  you  didn't 
acquire  the  right  data.) 

(2)  Data  are  in  hand.  These  data  include  results  of 
operational  testing,  and  whatever  other  data  are  opera¬ 
tionally  pertinent. 

(3)  These  data  are  examined  to  determine  if  the  sys¬ 
tem,  when  it's  operating  the  way  it's  supposed  to,  has  the 
capability  to  perform  the  necessary  missions  —  i.e.,  to 
determine  if  the  objectives  and  evaluation  criteria  asso¬ 
ciated  with  operational  effectiveness  have  been  met.  In 
this  process,  test  results  and  other  data  are  interpreted  in 
an  operational  framework  —  passed  through  an  operational 
filter  in  the  OTD's  head  —  to  arrive  at  their  operational 
meaning. 


(4)  Other  data  are  examined  to  determine  (basically) 
what  the  odds  are  that  the  system  will  operate  the  way  it's 
supposed  to  --  i.e.,  reliability,  maintainability,  and  the 
other  elements  of  operational  suitability.  Again,  the  OTD's 
operational  knowledge  and  experience  provide  a  filter  for 
the  data. 


Reminders : 

(1)  Don't  lose  sight  of  the  objectives. 

(2)  Think  systems  and  operational  missions 


(3)  Present  results  m  meaningful  operational  terms 
—  shun  the  purely  technical. 

(4)  Concentrate  on  what  it  will  do  as  it  is  --  it’s 
the  DA's  responsibility  to  figure  out  why  it  did  that  bad 
thing,  and  how  to  fix  it. 

304.  OT-I  Versus  OT-I I 

a.  The  reason  for  OT-I  is  to  provide  CNO  a  recommenda¬ 
tion  regarding  the  Milestone  II  decision.  Because  the 
equipment  to  be  tested  in  OT-I  is  probably  far  from  the 
eventual  production  configuration,  it  is  not  usually  possi¬ 
ble  to  estimate  reliability,  maintainability,  or  availabil¬ 
ity  quantitatively  (MTBF,  MTTR,  A  ,  etc.).  A  quantitative 
estimate  of  operational  effectiveness  can  be  made  in  OT-I, 
however,  because  (in  order  for  OT-I  to  have  meaning)  the 
equipment  is  functionally  like  the  proposed  production 
version.  So  it  is  possible  to  estimate  its  capability 
numerically.  OT-I  is  extremely  important  —  it  is  usually 
required,  and  it  usually  requires  hands-on  operation  (not 
hands-on  maintenance)  by  fleet  personnel. 

b.  OT-I I  —  specifically  OPEVAL  —  always  requires 
hands-on  operation  and  maintenance.  It's  our  last  chance  to 
insure  quality  equipment  in  the  fleet,  and  the  only  way  to 
do  that  is  to  use  it  the  way  the  fleet  will.  (Note  that  in 
the  case  of  phased  OT-I I,  the  early  pre-OPEVAL  phases  are 
treated  essentially  as  in  OT-I,  except  that  the  equipment 
becomes  progressively  more  like  the  production  configuration, 
and  operational  suitability  becomes  more  quantifiable.) 

c.  Don't  underrate  OT-I.  Although  OT-I I  may  appear 
more  glamorous,  many  other  considerations  (e.g.,  dollars 
already  invested)  may  override  COMOPTEVFOR ' s  OT-I I  recommen¬ 
dations.  The  greatest  opportunity  for  COMOPTEVFOR  to  influ¬ 
ence  future  fleet  equipment  (design,  performance,  and 
survivability)  is  as  a  result  of  thorough,  thoughtful  OT-I. 


305.  The  Philosophy  of  OT&E 


a.  The  following  extract  from  a  COMOPTEVFOR  document  is 
a  statement  of  COMOPTEVFOR  philosophy  of  operational  evalua¬ 
tion: 


Prior  to  OPEVAL,  a  new  weapons  system  should  have 
thoroughly  proven  its  capability  to  meet  technical  specifi¬ 
cations,  through  DT&E  culminating  in  TECHEVAL.  It  is  then 
COMOPTEVFOR * s  responsibility  to  structure  and  conduct  an 
OPEVAL  (at  minimum  cost  in  dollars  and  time,  commensurate 
with  risk  reduction)  that  will  prove  the  weapons  system's 
capability  in  a  realistic  operational  environment,  when 
maintained  and  operated  by  sailors,  subjected  to  routine 
wear-and-tear,  and  employed  in  typical  combat  conditions 
against  a  simulated  enemy  who  fights  back.  The  purpose  of 
OPEVAL  is  to  allow  an  accurate  assessment  to  be  made  of  the 
true  operational  effectiveness  and  operational  suitability 
of  the  weapons  system  in  actual  fleet  use  and  combat  employ¬ 
ment.  While  TECHEVAL  deals  principally  with  instrumented 
tests  and  statistically  valid  data,  OPEVAL  should  deal  with 
operational  realism  and  the  uncertainties  of  combat. 

Efforts  should  be  made  to  expose  the  weapons  system  to  as 
many  real-world  operational  circumstances  and  scenarios  as 
possible.  The  objective  is  not  always  to  acquire  statistic¬ 
ally  significant  data,  or  a  box  score  of  successes  and 
failures  (since  replications  are  seldom  possible),  but 
rather  to  gain  the  most  complete  understanding  possible  of 
the  weapons  system's  capabilities  under  stress.  In  techni¬ 
cal  testing,  it  is  generally  possible  to  state  the  purpose 
of  the  test  with  certainty.  In  operational  testing,  the 
principal  value  derived  is  often  unplanned,  resulting  not 
from  the  basic  purpose  of  the  test,  but  from  realistic 
aspects  that  were  injected  simply  because  they  are  likely  to 
exist  in  actual  fleet/combat  employment.  Thus  operational 
testing  is  more  an  art  than  a  science,  and  reasonable  oppor¬ 
tunity  should  be  provided  in  test  planning  for  the  unexpec¬ 
ted  to  occur  (as  it  usually  does  in  combat). 

b.  The  philosophy  of  OT&E  was  also  discussed  in  a 
COMOPTEVFOR  message  on  the  GP  (Guided  Projectile)  program. 
This  discussion  was  as  follows: 

To  the  maximum  degree  possible,  COMOPTEVFOR  desires 
to  minimize  both  the  time  and  cost  of  GP  T&E.  The  soundness 
of  the  acquisition  program  must  not  be  compromised,  however, 
and  a  sound  program  requires  both  DT&E  and  OT&E.  Funds 
could  be  saved  if  some  GP  test  firings  could  satisfy  both 
DT&E  and  OT&E.  While  this  may  be  possible  in  some  cases. 


the  widely  differing  objectives  of  GP  DT&E  and  OT&E  make  it 
unlikely  that  much  can  be  done  here.  Subparagraphs  (1)  and 
(2)  below  discuss  the  rationale  for  this: 

(1)  During  DT&E,  firing  is  properly  conducted  on  a 
round-by-round  basis,  with  each  shot  designed  to  test  some 
individual  specification  or  parameter  (e.g.,  the  number  of 
"G's"  pulled  by  the  projectile)  with  other  parameters  held 
constant.  The  test  is  designed  to  measure  technical  per¬ 
formance  of  the  system. 

(2)  In  OT&E,  proper  technical  performance  as  regards 
individual  specifications/parameters  is  assumed.  The  mission 
of  COMOPTEVFOR  is  to  assess  whether,  given  this  technical 
performance,  the  weapons  system  can  be  operationally  effec¬ 
tive  and  operationally  suitable  when  employed  under  typical 
combat  and  environmental  conditions,  by  fleet- type  personnel, 
against  an  enemy  who  fights  back.  Thus  OT&E  is  conducted  on 
a  mission-by-mission  basis,  varying  such  factors  as  sea 
state,  visibility,  own  ship  speed  and  maneuvers,  the  method 
of  illumination,  range,  firing  doctrine,  target  maneuvers, 
enemy  countermeasures ,  etc . 


Section  4 


Role  of  the  OTD 


401.  Function.  As  the  Navy's  independent  agent  for  OT&E, 
COMOPTEVFOR  is  charged  (OPNAVINST  5440.47)  with: 


Estimating  the  projected  operational  effectiveness 
and  operational  suitability  of  weapons  systems. 

b.  Developing  tactics  associated  with  new  weapons 
systems.  (See  Glossary  for  definitions  of  future ,  new,  and 
existing  weapons  systems . ) 

c.  Advising  CNO  on  the  adequacy  of  planned  T&E  to 
support  development  and  acquisition  decisions. 

In  his  assigned  area(s)  of  responsibility,  the  OTD  functions 
for  COMOPTEVFOR  in  the  detailed  planning,  the  supervision  of 
testing,  and  the  analysis  and  evaluation  of  test  results. 
These  functions  are  highlighted  in  more  detail  below,  not 
necessarily  in  the  order  in  which  the  OTD  will  perform  them, 
but  more  or  less  in  their  order  within  a  given  project. 

They  are  discussed  in  detail  in  later  sections  of  this 
Guide. 

402.  Planning.  The  OTD  plays  a  role  in  two  fundamental 
categories  of  planning:  long-range,  management  oriented; 
and  short-range,  test  oriented. 

a.  Long-range  planning  involves  providing  the  COMOP¬ 
TEVFOR  inputs  to  management- level  program  documents  (ORs, 
DCPs,  TEMPs,  etc.).  In  order  to  provide  these  inputs,  the 
OTD  should  be  able  to  answer  the  following  questions  as 
they  relate  to  his  project: 

( 1 )  From  an  operational  viewpoint,  why  develop  it? 

(2)  How  will  it  be  used?  In  what  installations? 

In  what  environments  (natural  and  manmade)? 

(3)  What  defines  its  operational  effectiveness?  What 
must  it  do? 

(4)  How  well  must  it  do  it?  When? 

(5)  What  must  DT&E  and  OT&E  do  to  prove  it  does  it? 

When? 
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(6)  What  are  the  appropriate  measures  of  operational 
suitability? 

(7)  What  numbers  are  required  for  these  measures? 

When? 

(8)  What  must  DT&E  and  OT&E  do  to  demonstrate 
them?  When? 

(9)  What  resources  are  required  for  OT&E? 

The  hardest  part  of  this  process  is  figuring  out  the  essen¬ 
tial  elements  of  operational  effectiveness  and  operational 
suitability.  A  contributing  factor  to  the  difficulty  in 
defining  these  elements  is  the  fact  that  a  number  of  sources 
or  agencies  may  appear  to  be  helping  (and  they  think  they 
are),  when  they're  not,  simply  because  they  don't  think  the 
way  we  have  to.  If  a  DA  provides  a  tentative  list  of 
"Required  Operational  Characteristics"  in  a  first-draft 
TEMP,  invariably  these  turn  out  to  be  "Required  Technical 
Characteristics"  —  because  he  thinks  technically,  not 
operationally.  And  if  you’re  not  constantly  alert  to  the 
danger,  you  can  make  the  same  error.  Don't  let  your  tech¬ 
nical  background  smother  your  operational  background. 
Confronted  with  a  new  weapons  system  or  equipment,  and 
having  understood  why  it's  being  developed,  ask  yourself 
(and  ask  your  analyst): 

(1)  What  must  it  do  from  an  operational  viewpoint? 

(2)  What  must  it  not  do  from  an  operational  view¬ 
point? 

For  example,  consider  a  buoy  carried  externally  on  a  subma¬ 
rine,  designed  to  release  automatically  if  test  depth  is 
exceeded,  surface,  and  transmit  at  regular  intervals  over 
the  life  of  its  battery  an  emergency  message  identifying  the 
submarine,  reporting  its  location  at  buoy  release,  etc. 

There  are  two  fundamental  characteristics  associated  with 
operational  effectiveness  of  the  buoy  (viewed  as  part  of  an 
overall  system  —  this  viewpoint  is  crucial  to  the  process): 

(1)  If  test  depth  is  exceeded,  there  must  be  a 
high  probability  that  an  accurate  distress  message  will  be 
received  and  acted  upon  at  the  ground-station. 

(2)  The  buoy  must  not  release  when  it’s  not  supposed 
to  (e.g.,  during  high-speed  transits,  maneuvers,  etc.). 


Note  that  parameters  such  as  output  power,  battery  endur¬ 
ance,  etc.,  while  related  to  the  first  operational  charac¬ 
teristic,  are  in  fact  technical  characteristics. 


If  the  elements  of  operational  effectiveness  and  operational 
suitability  are  defined  correctly,  then  the  rest  of  the  job 
becomes  almost  bookkeeping.  If  the  definition  is  wrong,  the 
error  may  remain  through  test  planning  and  test  operations , 
only  to  be  recognized  in  the  reporting  process  —  and  lead 
to  a  limitation  to  scope  which  says  we  didn't  ask  the  right 
questions . 

b.  Short-range  planning  involves,  primarily,  developing 
the  Test  Plan,  and  the  usually  undocumented  contingency 
plans  to  cover  unusual  circumstances.  These  last,  which 
frequently  exist  only  in  the  OTD's  head,  can  make  the 
difference  between  successful  and  unsuccessful  test  opera¬ 
tions.  They  are  created  by  an  OTD  who  asks  himself,  "What 
if,"  and  thinks  it  through. 

403 .  Supervising  the  Test 

a.  Make  sure  all  hands  know  what  they're  supposed  to 
do,  and  when. 

b.  Make  sure  data  are  collected  and  turned  in. 

c.  Be  prepared  to  alter  operations  if  unusual  circum¬ 
stances  warrant. 

d.  Keep  COMOPTEVFOR  advised. 

e.  Prevent  unauthorized  tampering  with  equipment  (this 
might  invalidate  test  data). 

404 .  Analysis  and  Evaluation  of  Test  Results 

a.  There  are  two  basic  functions  here.  Analysis  involves 
reconstruction  of  the  operational  situation  during  testing, 
and  deriving  the  various  measures  of  equipment  performance 
in  that  situation.  Evaluation  involves  the  thought  process 
of  relating  these  measures  to  the  objectives  in  order  to: 

(1)  Convince  decision-makers  regarding  operational 
effectiveness  and  operational  suitability  (at  CEB  and  DSARC 
briefings  (see  Glossary  for  definitions)  and  in  Evaluation 
Reports ) . 


(2)  Provide  useful  information  to  potential  users 
of  the  equipment  (through  Tactics  Guides). 

(Note:  The  Evaluation  Report  and  the  Tactics  Guide  are  the 
two  products  of  OT&E.) 

b.  Analysis  techniques  come  in  many  varieties  that  are 
individually  applicable  depending  on  the  type  of  data,  type 
of  equipment,  etc.  Selection  of  appropriate  techniques  is 
the  forte  of  the  Project  Analyst.  Interpretation  is  the 
OTD's  forte.  It  is  the  process  by  which  he  applies  his 
filter  of  operational  experience  to  test  data,  to  determine 
what  they  really  mean  from  an  operational  viewpoint. 

405.  Other  Duties  As  Directed.  The  OTD  is  the  primary 
source  of  information  on  all  aspects  of  his  project.  He 
briefs  at  high  (Flag)  levels,  he  answers  questions,  he 
drafts  responses  to  incoming  messages,  letters,  etc.  relat¬ 
ing  to  his  project.  The  more  he  knows  about  his  project, 
the  easier  his  job  is.  If  he  knows  everything  about  it, 
it's  still  not  easy.  Keep  the  Commander  informed  (through 
memos,  trip  reports,  requests  to  brief,  etc.).  Don't  let 
him  be  surprised! 

406.  Lessons  Learned.  When  you  have  made  a  mistake,  or  when 
things  have  gotten  fouled  up  for  some  other  reason,  tell 
your  Section  Head  or  ACOS.  There  is  a  possibility  that 
someone  else  may  make  the  same  mistake,  etc.,  and  perhaps 
that  can  be  avoided.  Your  Section  Head  and  ACOS  can  insure 
wide  distribution  of  lessons  learned  through  your  experi¬ 
ence,  and  maybe  others  can  profit  from  them.  The  same 
thinking  applies  to  good  things — if  you  learn  that  a  parti¬ 
cular  center  or  Laboratory  has  a  new,  proven  capability  (for 
example ) ,  describe  it  to  your  Section  Head  or  ACOS  and  let 
them  decide  if  other  Sections/Divisions  ought  to  know  about 
it. 

407 .  Overseeing  Preparation  and  Staffing  of  Test  Plans,  Evaluation 
Reports,  and  Tactics  Guides 

a.  General .  In  the  Headquarters,  the  OTD  (or  OTC,  when  the 
OTD  is  not  assigned  to  the  Headquarters)  is  responsible  for: 

(1)  Getting  draft  Test  Plans,  Evaluation  Reports,  and 
Tactics  Guides  prepared  for  staffing  and  introducing  these  drafts 
into  the  staffing  chain  (typically  Section  Head,  ACOS,  02,  01,  00). 

(2)  Getting  smooth  Test  Plans,  Evaluation  Reports,  and 
Tactics  Guides  prepared  for  signature  and  printing. 

b.  Draft  Documents 
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(1)  Draft  Test  Plans,  Evaluation  Reports,  and  Tactics  Guides 
are  prepared  (usually  by  WPC  (Word  Processing  Center))  in  double-space 
format. 


(2)  For  staffing  purposes,  draft  Test  Plans  and  Evaluation 
Reports  are  routed  in  specially  printed  manila  envelopes  obtainable 
from  the  Division  Admin  Office.  Draft  Tactics  Guides  are  routed 

in  Division  routing  folders. 

(3)  Draft  Test  Plans  are  routed  with  the  associated  TEMPs; 
draft  formal  Evaluation  Reports  are  routed  with  any  Quick-look 
Reports  that  preceded  them.  OTDs/OTCs  may  include  such  other 
background  material  that  they  consider  necessary. 

(4)  Original  art  work  and  other  irreplaceable  material 
(e.g.,  original  reports  from  Commanding  Officers  of  project  ships) 
are  not  routed  with  drafts  —  use  the  Xerox  machine. 

c.  Documents  for  Signature 

(1)  Smooth,  for-signature  documents  are  routed  to  the  signer 
(in  the  case  of  00,  via  02)  in  ready- for-printing  form. 

(2)  "Ready- for-printing”  means: 

(a)  Original  (or  original  quality)  drawings  and  typed 

material. 


(b)  Glossy  prints  of  photos. 

If  in  doubt  about  printability,  ask  the  Print  Shop  Manager  —  early. 
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Section  5 


Role  of  the  OTC 


501.  Function 


a.  For  projects  with  an  OTD  assigned  from  within  the 
Headquarters,  the  OTD  and  OTC  may  be  the  same  person,  or  the 
OTC  may  act  in  a  normal  section  head  capacity  (as  the  OTD's 
day-to-day  supervisor,  who  acts  for  him  in  his  absence). 

For  some  large  projects  (e.g.,  ship  evaluation),  the  OTC  may 
be  from  one  Warfare  Division,  with  a  team  of  OTDs  from  other 
Divisions . 

b.  For  projects  with  an  OTD  assigned  from  Deputy,  a 
Squadron,  or  a  Detachment,  the  OTC's  functions  are  different. 

(1)  He  provides  guidance  on  the  project  through  the 
OTD's  organizational  structure.  Guidance  --  not  direction. 
Disagreements  are  resolved  at  parallel  higher  levels. 

(2)  He  provides  the  interface  with  Washington 
(including  negotiations  on  project  matters). 

(3)  He  is  the  Headquarters  focal  point  on  project 
matters,  and  initiates  all  project-related  staffing  within 
the  Headquarters. 

502 .  Project  Assignment  and  Reassignment 

a.  To  assist  in  tracking  T&E  and  RDT&E  support,  CNO 
(OP-098)  assigns  a  T&E  number  to  each  acquisition  program. 
This  number  continues  for  the  life  of  the  program,  and,  for 
ACAT- I ,  II,  and  III  programs ,  is  the  TEMP  number . 

b.  Before  a  new  T&E  number  is  assigned,  OP-983  alerts 
the  DCOS,  who  alerts  the  appropriate  Warfare  Division.  The 
ACOS  names  an  OTC  (and  an  OTD  if  the  project  will  be  prose¬ 
cuted  by  the  Headquarters).  This  information  is  relayed  by 
the  DCOS  to  OP-983,  who  prepares  the  formal  letter  assigning 
the  new  T&E  number.  This  letter  directs  preparation  of  TEMP 
(ACAT- I,  II,  and  III  programs),  and  lists  the  OPNAV,  NAVMAT, 
and  OPTEVFOR  points  of  contact. 

c.  Projects  are  either  prosecuted  by  the  Headquarters 
or  are  reassigned,  usually  to  Deputy  or  an  AIRTEVRON,  with 
the  Commander's  approval.  The  decision  on  prosecution  (and 
hence  location  of  the  OTD)  is  based  on  considerations  such 


(1)  Availability  of  a  qualified  OTD. 

(2)  Test  platform  requirements  (e.g.,  type  aircraft) 

(3)  Requirements  to  use  specific  test  facilities 
(e.g.,  ranges ) . 

Usually  if  a  project  is  reassigned  initially,  it  remains  so 
until  it  is  terminated.  Occasionally,  however,  considera¬ 
tions  such  as  those  just  mentioned  dictate  that  specific 
phases  of  a  project  (e.g.,  OPEVAL)  be  prosecuted  by  differ¬ 
ent  agencies. 

d.  When  the  Commander  has  approved  a  reassignment,  the 
cognizant  OTC  prepares  the  formal  COMOPTEVFOR  reassignment 
letter.  This  letter  provides  details  on: 

(1)  Program  documentation  that  is  available. 

(2)  Coordination  and  liaison  authority. 

(3)  Signature  authority  for  test  plans. 

(4)  Reports  (including  partial  and  quick-look)  that 
will  be  required. 

(5)  Program  structure  —  i.e.,  T&E  versus  mile¬ 
stones  . 

e.  Subsequent  to  project  reassignment,  all  project  cor¬ 
respondence  is  provided  the  OTD,  either  automatically  by 
COMOPTEVFOR  Admin  or  by  message  readdressal  (OTC's  respon¬ 
sibility).  The  OTC  also  ensures  that  all  information/deci¬ 
sion  memos  to  the  Commander  are  forwarded  to  the  OTD. 


Section  6 


Program  Structure 

601.  Definition.  Program  structure  is  the  relationship  be- 
tween  the  RDT&E  profile  and  the  production  profile  (see 
Figure  6-1).  Proper  program  structure  insures  that  both 
DT&E  and  OT&E  inputs  will  be  available  to  decision  makers  at 
program  milestones.  This  requires  a  properly  phased  T&E 
program,  supported  by  adequate  test  articles  (with  defined 
performance  thresholds ) . 

602 .  OTD/OTC  Guidelines  Regarding  Program  Structure 

a.  If  OPTEVFOR's  position  will  be  that  OT&E  is  required, 
make  sure  a  new  program  is  designated  ACAT-III  or  higher 
(see  OPNAVINST  3960.10  for  ACAT  definitions  and  criteria). 
ACAT-III  or  higher  means  a  TEMP  is  required 

b.  Concentrate  on  the  TEMP  —  do  everything  you  can  to 
make  it  brief,  factual,  and  clear . 

c.  Lay  out  the  RDT&E  profile.  Identify  the  major 
milestones. 

d.  Focus  on  testing  tiat  supports  Milestone  II  (FSD 
(full-scale  development)).  This  may  be  the  last  time  OPTEV- 
FOR  can  influence  major  system  charateristics . 

e.  Identify  key  development  risk  areas  (technical  and 
operational).  Insure  that  the  proposed  DT-I/OT-I  phases 
will  demonstrate  that  engineering  is  reasonably  complete; 
that  all  significant  design  problems  (including  reliability, 
maintainability,  compatibility,  interoperability,  and  logistical 
considerations)  have  been  identified;  and  that  solutions  to 
the  above  problems  are  in  hand. 

f.  Decide  on  the  need  for  at-sea  OT-I.  It  may  be 
expensive  and  time-consuming,  and  may  be  resisted  by  the  DA. 

But  if  the  development  requires  it  —  if,  for  instance, 
performance  in  a  Task  Force  environment  is  a  critical  issue  — 
early  at-sea  testing  can  save  both  development  costs  and 

time  in  the  long  run. 

g.  Identify  test  articles  clearly  (by  nomenclature 
(production  prototypes,  etc.),  number,  availability  date, 
and  usage  (OT  phase)). 


PROGRAM  STRUCTURE 


PRODUCTION 


h.  Clearly  define  the  production  profile  —  in  terns 
of  nomenclature  by  lots,  numbers  to  be  produced,  long-lead 
requirements,  and  funding  requirements. 

i.  Insure  that  production  lots  are  released  only  at 
Milestone  III  points  —  and  that  these  points  are  preceded 
by  phases  of  OT-II. 

j.  structure  OT-II  carefully  and  insure  that  OPEVAL: 

(1)  Is  a  realistic  simulation  of  combat. 

(2)  Is  long  enough  —  particularly,  to  demonstrate 
reliability. 

(3)  Uses  production- representative  test  articles. 

(4)  Uses  operational  thresholds  adequate  for  fleet 
useage.  (Specify  simple,  dominant  thresholds.) 

k.  Specify  projected  FOT&E  (OT-III  and  OT-IV). 


603.  Do's  and  Don 1 t 1 s  Regarding  Program  Structure 

a.  Don't  wait  until  the  program  has  been  established. 
Get  together  with  the  DA  early  and  help  him  shape  program 
structure.  The  earlier  the  better. 

b.  Don't  tell  the  DA  to  develop  a  rough  draft  (of  DCP, 
TEMP,  etc.)  for  our  comment.  Help  him  draft  it  (see  Section 
7).  If  he's  got  a  Navy  lab  or  a  consulting  contractor 
working  on  it,  volunteer  to  help  them. 

c.  Do  draft  a  "program  structure"  letter  or  message  as 
early  as  possible  --  a  brief  (one  page  of  text,  one  chart 
like  Figure  6-1)  outline  of  program  structure  fundamentals. 

d.  Do  spend  extensive  effort  on  test  thresholds.  Work 
with  the  sponsor  and  the  DA.  Specify  operational  thresholds 
for  OT&E .  (Note  that  C0M0PTEVF0R  establishes  the  measures  - 
the  actual  threshold  values  amount  to  recommendations  to 
CNO;  he  must  ultimately  approve.)  Don't  ignore  technical 
thresholds  —  make  sure  they' re  consistent  with  operational 
thresholds. 

e.  Do  define  projected  FOT&E  early,  and  in  detail, 
since  FOT&E  will  always  be  required,  unless  no  unresolved 
issues  remain  from  IOT&E.  Clarify  the  funding  at  the  outset 
(don't  let  legitimate  OT-III  be  moved  into  OT-IV). 


a.  Policy  and  procedures  for  ASU/PASU  are  contained  in 
OPNAVINST  4720.9.  Each  OTD/OTC  should  be  familiar  with  the 
current  issue  of  this  instruction. 

b.  COMOPTEVFOR ' s  recommendation,  based  on  OT&E,  is  con¬ 
sidered  in  every  ASU/PASU  decision  for  ACAT-I,  II,  or  III 
systems.  This  recommendation  may  be  briefed  at  a  program 
review  (as  outlined  in  OPNAVINST  3960.10),  and  is  provided 
in  each  evaluation  report  of  OPEVAL.  In  addition,  CNM  (MAT 
04),  as  a  matter  of  routine,  requests  COMOPTEVFOR ' s  recom¬ 
mendation  on  each  ASU/PASU  decision  being  considered  in 
NAVMAT  (including  ACAT-IV  decisions).  Note  that  a  MAT  04 
request  may  be  dated  a  day  or  a  week  after  he  has  received 
our  OPEVAL  report  containing  an  ASU/PASU  recommendation.  He 
isn't  asking  because  he  didn't  read  his  mail;  he's  asking 
because  by  CHNAVMAT/COMOPTEVFOR  agreement,  we  will  always  be 
asked  specifically  when  a  decision  is  being  made,  to  obtain 
our  best  judgment,  current  at  the  time  of  the  decision. 

This  allows  for  the  rare  case  when  our  recommendation  regard¬ 
ing  a  system  changes  after  the  OPEVAL  report  has  been  pub¬ 
lished,  or  when  the  ASU  request  includes  configurations  or 
components  that  were  not  tested  during  OPEVAL. 

c.  COMOPTEVFOR  responses  to  MAT  04  requests  for  comments 
on  proposed  ASU  are: 

(1)  Prepared  in  message  format,  as  shown  in  the 
following  pages,  under  the  direction  of  the  cognizant  ACOS. 

(2)  Reviewed  by  the  DCOS  and  the  Chief  of  Staff 
prior  to  submission  to  the  Commander. 

( 3 )  Approved  by  the  Commander . 

(4)  Released  within  2  working  days  of  receipt  of  the 
MAT  04  request,  unless  unusual  circumstances  prohibit. 


GUIDE  FOR  MESSAGES  COMMENTING  ON 


PROPOSED  APPROVAL  FOR  SERVICE  USE 


1.  The  message  is  usually  in  three  paragraphs,  as  shown  in  the 
accompanying  incomplete  sample. 

2.  Paragraph  1  is  the  reason  for  the  message.  Use  the  term  "provi¬ 
sional"  (in  the  subject  and  in  this  paragraph)  if  provisional  approval 
is  under  consideration. 

3.  Paragraph  2  is  the  COMOPTEVFOR  recommendation  on  the  issue  in 
question. 

4.  Paragraph  3  is  a  brief  summary  of  the  rationale  for  the  COMOPTEVFOR 
recommendation,  citing  appropriate  COMOPTEVFOR  tests,  reports,  etc. 

The  major  results  of  any  OT&E  conducted  but  not  yet  formally  reported 
should  be  included.  If  the  COMOPTEVFOR  recommendation  is  for  approval, 
this  paragraph  should  state  that  approval  was  recommended  in  the 
appropriate  COMOPTEVFOR  report,  or  that  approval  is  recommended  based 
on  the  system's  having  met  the  criteria  of  OPNAVINST  4720.9.  If  the 
COMOPTEVFOR  recommendation  is  to  not  approve  the  item  for  service  use, 
this  paragraph  should  identify  specifically  the  criteria  of  OPNAVINST 
4720.9  that  have  not  been  satisfied.  Furthermore,  this  paragraph 
should  specify  the  steps  (additional  testing,  etc.)  necessary  to 
support  a  COMOPTEVFOR  recommendation  for  approval. 
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FROM:  COMOPTEVFOR  NORFOLK  V A 
TO:  CHNAVMAT  WASHINGTON  DC 
INFO  CNO  WASHINGTON  DC 

COMNAV . SYSCOM  WASHINGTON  DC 

CLASS IF ICATI0N//N03TL0// 

SUB J ’  {PROVISIONAL}  APPROVAL  FOR  SERVICE  USE  OF . 

A.  CHNAVMAT  WASHINGTON  DC  DATE/TIP1E  GROUP 

B-  COMOPTEVFOR  LTR  SER . OF . 

C.  OPNAVINST  4720. ID 

1.  {*>  REF  A  REQUESTED  COflOPTEVFOR  COMMENT  ON  FORTHCOMING  DECISION 

ON  {PROVISIONAL}  APPROVAL  FOR  SERVICE  USE  FOR . 
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AS  REQUIRED  REF  C.  RECOMMEND  {PROVISIONAL}  APPROVAL  FOR  SERVICE  USE 
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Section  7 


COMOPTEVFOR  Contributions  to  Program  Documents 

701.  Introduction 

a.  COMOPTEVFOR  is  required  by  his  charter,  both  explicitly 
and  implicitly,  to  advise  CNO  not  only  on  the  capability  of  new 
weapons  systems,  but  also  on  the  adequacy  of  performance  specifi¬ 
cations  and  the  adequacy  of  all  planned-  T&E  to  support  development 
and  acquisition  decisions.  Notice  that  "all  planned  T&E"  doesn't 
mean  just  OT&E,  and  that  "adequacy  of  all  planned  T&E"  means 
adequacy  of  opportunity  for  T&E  at  proper  times  and  in  adequate 
amounts  to  provide  CNO  with  information  pertinent  to  decision-making. 
So,  when  you  help  make  a  program  document  more  nearly  perfect, 
you're  responding  to  a  primary  task  of  COMOPTEVFOR.  The  task  is 
important;  it  deserves  your  best  shot.  Your  influence  on  the 
entire  development  and  deployment  of  a  new  weapons  system  can  never 
again  be  as  great  after  the  first  NDCP  is  signed  with  a  faulty 
development  and  acquisition  strategy  (program  structure,  as  discussed 
in  Section  6)  embedded  in  it. 

b.  COMOPTEVFOR  contributes  to  program  documents  in  two  basic 

ways:  informally,  at  the  working  level,  when  the  OTD/OTC  assists 

in  preparation  of  early  drafts;  and  formally,  in  comment  letters. 
While  COMOPTEVFOR  is  interested  in  the  entire  content  of  each 
program  document,  some  areas  of  these  documents  are  of  more  interest 
than  others.  These  are  discussed  below. 

702.  ORs  (Operational  Requirements) .  The  format  and  content  of 
an  OR  is  described  in  OPNAVINST  5000.42.  Be  familiar  with  this 
instruction  before  reviewing  an  OR.  Concentrate  your  review  on 
Section  I,  Operational  Need  (emphasis  on  the  operational  problem) ; 
Section  II,  Operational  Concept;  and  Section  III,  Capabilities 
Required  (emphasis  on  operational  parameters/criteria  and  operational 
development) .  Although  your  comments  will  usually  be  confined  to 
Section  III,  you  may  recommend  changes  to  any  section  so  long  as 

you  provide  operational  rationale  for  the  changes. 

703.  DPs  (Development  Proposals) .  These  documents  are  also 
addressed  in  OPNAVINST  5006.42. Concentrate  this  review  on  Section 
IV,  Program  Alternatives;  Section  V,  Effectiveness  and  Cost  Com¬ 
parison  of  Alternatives  (emphasis  on  T&E  aspects  of  proposed 
development/production  schedules);  Section  VI,  Risks  (emphasis  on 
uncertainties  to  be  resolved);  Section  VII,  Test  and  Evaluation; 
Section  VIII,  Other  Factors;  and  Section  XI,  The  Development  Plan(s), 
Achievement  Milestones  and  Thresholds.  Your  comment  will  usually 
address  the  adequacy  of  planned  T&E  to  support  scheduled  program 
milestones. 
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704.  TLRs  (Top-Level  Requirements)  and  TLSs  (Top-Level  Specifi¬ 
cations)  .  For  ship  development  and  ship  acquisition  programs, 

TLRs  and  TLSs  are  prepared  after  the  OR  or  DP.  This  additional 
set  of  documents,  discussed  in  OPNAVINST  9010.300,  is  necessary 
because  of  the  length  and  complexity  of  the  ship  design  process. 

Your  review  and  comments  will  follow  the  guidelines  discussed  above 
for  ORs  and  DPs. 

705.  DCPs  (Decision  Coordinating  Papers)  and  NDCPs  (Navy  DCPs) . 

These  documents  are  discussed  in  DODINST  5000. 2  and  OPNAVINST 
5000.46.  Review  the  entire  document,  but  pay  particular  attention 
to  Annex  A  (Goals  and  Thresholds)  to  ensure  that  the  performance 
parameters  and  supportability  and  manpower  parameters  are  complete 
and  unambiguous,  and  that  the  values  associated  with  them  are 
accurate  and  consistent. 

706.  TEMPs .  TEMPS  are  discussed  in  detail  in  OPNAVINST  3960.10, 
and  are  the  subject  of  Section  8  of  this  Guide.  Draft  TEMPs  may 
be  reviewed  in  their  entirety  twice;  once  when  the  DA  gives  us  a 
completed  draft,  and  again  during  CNO  (OP-098)  review.  Before  the 
first  review,  you  should  have  provided  the  DA  with  Required 
Operational  Characteristics  of  Part  I,  OT&E  schedule  inputs  for 
Part  II,  a  complete  Part  IV,  and  resource  requirements  for  Part  VI. 
Your  reviews  of  the  complete  TEMP  should  address  all  parts,  even 
including  your  own  draft  Part  IV.  Inadequacies  in  DT&E  are  important 
both  to  you,  in  that  they  might  inhibit  good  operational  testing, 
and  to  OP-983,  who  must  advise  on  all  T&E.  In  this  regard,  OP-983 

is  you.  You  should  be  especially  sensitive  to  resource  and  schedule 
inadequacies  in  the  final  draft  TEMP,  and  ensure  that  COMOPTEVFOR 
points  them  out  to  CNO. 

707 .  PEPS  (Program  Elements  Descriptive  Summaries] l/CDSs 
(Congressional  Data  Sheets) .  These  documents  are  described  in  the 
DOD  Budget  Guidance  Manual  (NAVCOMPTINST  7102.1),  and  are  prepared 
annually  by  the  DA.  COMOPTEVFOR  reviews  drafts  of  these  documents, 
and  provides  the  OT&E  write-ups  in  their  T&E  sections.  Guidance  is 
promulgated  by  the  DCOS  as  each  annual  cycle  begins. 

708.  MENS  (Mission  Element  Need  Statement) .  This  document  is 
required  by  DODINST  5000.2  to  justify  initiation  of  a  new  major 
system  acquisition.  It  is  prepared  by  the  individual  Service  and 
submitted  to  SECDEF  to  support  a  Milestone  0  decision.  We  review 
the  entire  MENS,  and  provide  suggested  improvements  where  possible. 

709.  Mini-NDCP.  By  OP-098  memo  ser  987/239830  of  2  Jul  1979,  a 
revised  NDCP  format  was  specified  for  ACAT  III  and  IV  programs — 
the  Mini-NDCP.  This  four-page  document  should  be  reviewed  in  its 
entirety,  with  particular  attention  to  page  2  (goals  and  thresholds) . 
(Note  that  COMOPTEVFOR  ltr  ser  1446  of  26  Oc  1979  forwarded  OPNAV 
guidance  for  Mini-NDCP  preparation.) 
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710.  IPSs  (Integrated  Program  Summaries) .  These  documents  are 
required  by  DODINST  5000.2,  which  specifies  their  format  and  content. 
Concentrate  your  review  on  the  following  topics:  System  Vulnerability 
Organizational  and  Operational  Concept,  Test  and  Evaluation, 
Reliability  and  Maintainability,  and  Computer  Resources. 

711.  Preparation  of  Comment  Letters.  This  paragraph  provides 
guidance  for  preparing  comment  letters,  and  comment  letter  format. 

Use  this  format  for  all  COMOPTEVFOR  comment  letters  unless  a 
different  one  is  prescribed  by  higher  authority  (e.g.,  DOD 
Directive  7650.2  for  commenting  on  GAO  reports). 

a.  Comment  letters  usually  consist  of  a  letter  and  an  enclosure 
(illustrated  in  the  next  few  pages) .  The  enclosure  to  a  comment 
letter  provides  specific  recommended  changes  to  the  document  being 
commented  on,  ordered  front-to-back,  plus  the  rationale  for  each 
recommended  change  Prepare  the  enclosure  first.  The  comment  letter 
emphasizes  the  key  point (s)  of  the  enclosure. 

b.  Before  commencing  your  comment  letter  draft,  study  the 
correspondence  thoroughly,  making  notes. 

c.  Don't  comment  on  typographical  or  other  minor  errors  that 
don't  affect  meaning.  Don't  comment  on  technical  parameters  that 
have  no  operational  significance. 
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DEPARTMENT  OF  THE  NAVY 

COMMANDER  OPERATIONAL  TEST  AND  EVALUATION  FORCE 
NORFOLK,  VIRGINIA  23511 

644 :skd 

3930 

Ser 


From:  Commander  Operational  Test  and  Evaluation  Force 

To:  Chief  of  Naval  Operations 

Sub j :  Navy  Decision  Coordinating  Paper  for  Development  of 

the  Combined  UHF/VHF  AM/FM  Radio  System  (NDCP  WCC  94) 

Ref:  (a)  CNO  ltr  ser  943/C607274  dtd  22  July  1976 

Enel:  (1)  Detailed  Comments  on  NDCP  WCC  94 

1.  Enclosure  (1)  to  reference  (a)  has  been  reviewed.  The 
major  comment  is  provided  below.  Detailed  comments  are 
contained  in  enclosure  (1) . 

2.  As  the  draft  is  written,  a  full-scale  development  decision 
(Milestone  II)  could  be  made  without  any  hardware  and  T&E 
(test  and  evaluation) .  While  the  radio  itself  is  considered 
low  risk,  the  antenna  is  not.  Adequate  T&E  is  require  prior 
to  Milestone  II,  and  the  NDCP  should  be  modified  to  include  it 


Copy  to : 

CNO  (OP- 986) 
(OP- 94  3) 


R.  E.  CRISPIN 
Chief  of  Staff 


Detailed  Comments  on  NDCP  WCC  94 


1.  Page  13,  par.  E.,  Program  Structure ,  Milestones ,  and 
Threshold  Events.  The  Milestone/Event  Chart  contained  in 
this  paragraph  needs  to  be  changed/updated  to  reflect  the 
program  structure.  The  recommended  structure  is: 


Milestone/Event 


Date 


Approve  NDCP 

Specifications,  requirements,  RFP 
TEMP 

Release  FY  77  RDT&E  Funds 

Contract  Award  for  Advanced  Development  Model 
(Milestone  I ) 

DT/OT-I  Testing 

Preliminary  Design/Program  Review 

Contract  Award  for  Engineering  Development  Model 

(Milestone  II) 

Start  Contractor  Tests 
Critical  Design/Program  Review 
TECHEVAL 
OPEVAL 

Critical  Design/Program  Review 
Approval  for  Service  Use 
Milestone  III 
IOC 


AUG  76 
SEP  76 
OCT  76 
OCT  76 

JAN  77 
MAR-JUN  77 
AUG  77 

SEP  77 
DEC  77 
DEC  77 
JAN-MAR  78 
MAY-AUG  78 
SEP  78 
OCT  78 
OCT  78 
AUG  79 


Rationale:  This  schedule  would  allow  for  more  develop- 
mental  testing  prior  to  Milestone  II  to  increase  confidence 
prior  to  TECHEVAL/OPEVAL. 

2.  Page  13  and  14,  par.  IX,  Test  and  Evaluation.  This 
section  should  be  rewritten  to  allow  for  testing  (DT/OT-I) 
prior  to  Milestone  II  and  further  testing/miles toning 
throughout  the  program. 

Rationale:  To  allow  for  sufficient  testing  to  increase 
confidence  at  each  Milestone. 


3.  Page  15,  par.  2.b.,  OPEVAL,  ’’Scope”.  Change  the  first 
sentence  to  read  as  follows: 


"Determine  operational  effectiveness  and  operational 
suitability  for  service  use.” 

Rationale :  COMOPTEVFOR  does  not  limit  assessment  to 
suitability  alone;  assessment  must  be  made  of  operational 
effectiveness  and  operational  suitability. 


I 


4.  Page  15,  par.  2.b.,  OPEVAL ,  11  Location" .  Change  to  read 
as  follows: 


"To  be  determined  as  services  become  available.  Present 
plans  call  for  evaluation  to  be  conducted  on  both  the  East 
and  West  Coasts . " 

Rationale:  Present  planning  requires  use  of  VX-4  and 
VX-5  on  the  West  Coast  for  two  phases  of  the  evaluation. 

Marine  Corps  assets  on  the  East  Coast  for  the  third  phase, 
and  fleet  assets  on  the  East  Coast  for  the  final  phase. 

5.  Page  15,  par.  2.b.,  OPEVAL,  "Duration".  Change  to  read 
as  follows: 

"Approximately  3  months  from  MAY  78  through  AUG  78 . " 

Rationale:  Planning  meetings  have  indicated  that  TECH- 
EVAL/OPEVAL  may  be  moved  forward. 

6.  Page  15,  par.  C.,  Table  of  DT&E  Performance  Objectives. 

This  paragraph  should  be  changed  to  the  format  contained  in 
OPNAVINST  5000.46. 

Rationale:  Standard  format  should  be  used. 

7.  Page  15,  par.  D.,  Table  of  OT&E  Performance  Objectives. 

The  below  table  is  recommended: 

Milestone  II  Milestone  III  Post  Milestone  III 

"Parameter  Threshold  Demons t.  Threshold  Demonst.  Goal  Demons t. 


1. 

RF  Power 

10W  (AM/FM) 

Same 

Same 

2. 

FM  Cap 

30-88  MHz/ 
225-400  MHZ 

Same 

Same 

3. 

AM  Cap 

116-156  MHZ/ 
225-400  MHZ 

Same 

Same 

4. 

MTBF 

500  hours 

1000  hours 

1000  hours 

5. 

MTTR 

2  hours 

1  hour 

1  hour 

6. 

Ao 

0.996 

0.998 

0.998" 

Rationale: 

This  table  sets 

thresholds  to  assess 

both 

operational  effectiveness  and  operational  suitability. 


7-7 

PAGE  7-8,  REVERSE,  BLANK 


Section  8 


The  TEMP,  and  Scheduling  RDT&E  Support 


801.  Importance  of  the  TEMP.  To  COMOPTEVFOR,  the  TEMP  is 
the  most  important  single  document  associated  with  an  acqui¬ 
sition  program.  It  summarizes  all  the  significant  aspects 
of  the  program  (including  costs,  schedules,  and  planned 
installations),  and  relates  the  three  types  of  T&E  (OT&E, 
DT&E,  and  PAT&E)  that  will  provide  the  measures  of  program 
progress.  Approval  constitutes  CNO  direction  to  fund  and 
execute  the  T&E,  and  is  a  contract  between  the  DA  and  COM¬ 
OPTEVFOR  on  this  T&E.  The  format  and  content  of  a  TEMP  is 
described  in  OPNAVINST  3960.10.  Each  OTD  and  OTC  must  be 
familiar  with  OPNAVINST  3960.10  in  detail. 

802 .  TEMP  Preparation 

a.  A  TEMP  is  prepared  by  the  DA  in  cooperation  with 
COMOPTEVFOR  (and  PRESINSURV  when  appropriate).  "In  coop¬ 
eration  with"  means  "With  active  participation  by."  COMOP¬ 
TEVFOR  contributes  to  all  parts  of  the  TEMP  (in  working 
sessions,  through  comment  letters,  etc.).  COMOPTEVFOR 
provides  some  parts  of  the  TEMP.  These  parts,  discussed 
later,  are  drafted  by  the  OTD  (not  the  DA,  Program  Sponsor, 
or  anyone  else ) . 

b.  According  to  OPNAVINST  3960.10,  a  TEMP  is  prepared 
early  in  each  new  acquisition  program,  and  is  approved  prior 
to  Milestone  I .  This  requires  that  the  OTD/OTC  know  what's 
going  on.  Pay  attention  to  program  documentation  (ORs, 

NDCPs ,  MIPs,  etc.),  and  arrange  to  talk  with  the  DA  when  you 
spot  a  new  development  starting.  Volunteer  to  help  with 
TEMP  preparation. 

c.  If  you  don't  think  TEMP  development  is  moving  fast 
enough  get  your  OTC,  Section  Head,  and  ACOS  involved  as 
necessary.  Don't  let  it  slide  until  it's  too  late! 

d.  Handle  as  much  as  you  can  at  informal  working  ses¬ 
sions,  and  through  informal  inputs.  But  always  make  sure 
the  DA  understands  that  a  formal  COMOPTEVFOR  chop  will  take 
place,  and  that  if  the  thing  isn't  the  way  we  want  it,  we'll 
say  so  officially. 

803.  COMOPTEVFOR  InDUts  To  TEMPs 


a.  COMOPTEVFOR  double-checks  the  entire  TEMP  with  par¬ 
ticular  attention  to  the  following: 
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(1)  Mission  (paragraph  1,  Part  I).  Check  for  accur¬ 
acy  and  completeness  from  an  operational  viewpoint  —  parti¬ 
cularly  with  respect  to  the  planned  operational  environment. 

(2)  Key  Functions  (paragraph  2a,  Part  I).  Ensure 
that  the  mission/ function  matrix  (if  one  is  included)  is 
complete . 

(3)  Interfaces  (paragraph  2b,  Part  I).  Ensure  com¬ 
pleteness  . 

(4)  Required  Technical  Characteristics  (paragraph  4, 
Part  I ) .  Ensure  that  performance  goals  and  thresholds  are 
consistent  with  Required  Operational  Characteristics.  For 
example,  a  threshold  MTBF  of  150  hours  is  not  consistent 
with  a  threshold  24-hour  mission  reliability  of  0.90  —  it 
would  have  to  be  nearly  230  hours. 

(5)  Management  (paragraph  1,  Part  II).  Ensure  that 
COMOPTEVFOR ' s  responsibilities  are  stated  clearly  and  accu¬ 
rately  —  particularly  with  respect  to  any  planned  combined 
development  and  operational  testing. 

(6)  DT&E  Outline  (Part  III).  Check  for  completeness 
(including  survivability/vulnerability),  and  relevance  to 
the  issues. 

b.  COMOPTEVFOR  prepares  the  Required  Operational  Char¬ 
acteristics  section  (paragraph  3,  Part  I)  of  the  TEMP.  The 
first  set  of  characteristics  (for  operational  effectiveness) 
state  plainly,  with  numbers  where  possible,  what  the  equip¬ 
ment  is  expected  to  do  (what  missions  or  essential  functions 
it  must  accomplish)  when  it's  working  the  way  it's  supposed 
to.  The  second  set  of  characteristics  (for  operational  suit¬ 
ability)  state  the  reliability,  maintainability,  etc.  char¬ 
acteristics  required  to  insure  that  it  will  work  the  way 
it’s  supposed  to.  If  CNO  has  not  specified  numbers  for 
these,  suggest  them  in  the  draft.  TEMP  approval  constitutes 
CNO  direction  regarding  the  numbers  in  the  approved  TEMP. 

c.  COMOPTEVFOR  prepares  the  Operational  Issues  (para¬ 
graph  5b,  Part  I). 

d.  COMOPTEVFOR  prepares  the  OT&E-related  portions  of 
the  Integrated  Schedule  (paragraph  2,  Part  II),  and  Part  VI 
(Special  Resource  Summary).  Use  Section  6  of  this  Guide 
when  preparing  inputs  to  paragraph  2,  Part  i  I. 
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e.  COMOPTEVFOR  prepares  Part  IV  (OT&E  Outline).  In 
Part  IV,  a  number  of  OT&E  phases  (OT-IIA,  OT-IIB,  OT-IIIA, 
etc. )  are  defined.  Each  is  created  for  a  specific  reason, 
usually  to  provide  information  necessary  to  an  important 
development  or  production  decision.  For  example,  early 
phases  of  IOT&E  often  are  designed  to  assist  in  deciding 
between  competing  system  designs.  Later  phases  can  be  de¬ 
signed  to  support  ASU  decisions.  FOT&E  is  intended  to  in¬ 
fluence  decisions  on  production  rates,  and  decisions  to  use 
a  system  against  a  new  threat  or  to  begin  development  of 
advanced  systems.  The  reason  for  each  phase  of  OT&E  should 
be  obvious  to  the  reader  of  Part  IV  of  a  TEMP.  That  is,  it 
should  be  obvious  to  the  reader  what  successful  accomplish¬ 
ment  of  each  phase  will  lead  to.  This  information  should 
appear  under  the  "OT&E  Objectives"  paragraph  associated  with 
each  phase,  and  it  should  be  the  first  information  in  that 
paragraph,  as  illustrated  below: 

"(2)  OT&E  Objectives.  Successful  accomplishment  of 
OT-II  will  allow  a  recommendation  on  ASU.  Specific  objec¬ 
tives  are : " 


"(2)  OT&E  Objectives.  Successful  accomplishment  of 
OT-II I  will  provide  confidence  in  the  design,  to  support  a 
decision  for  full-rate  production.  Specific  objectives  are:" 

804 .  Required  Operational  Characteristics 

a.  Among  the  various  tasks  performed  by  an  OTD/OTC, 
that  of  defining  a  system's  required  operational  character¬ 
istics  has  the  greatest  potential  for  long-range  signifi¬ 
cance  (good  and  bad).  For  these  characteristics  will  ulti¬ 
mately  define  the  objectives  (and  associated  evaluation 
criteria)  of  the  various  phases  of  OT&E.  If  these  charac¬ 
teristics  are  wrong,  then  the  objectives/criteria  of  OT&E 
will  be  wrong,  and  we  will  evaluate  the  wrong  things — we 
will  answer  the  wrong  questions.  For  this  reason,  deriva¬ 
tion  of  required  operational  characteristics  deserves  care¬ 
ful  thought.  Do  not  attempt  to  do  all  the  careful  thinking 
yourself — discuss  the  subject  with  as  many  intelligent 
people  as  you  can  find.  And  don't  restrict  them  to  experts 
in  the  particular  warfare  area — other  intelligent  people  may 
have  a  better  view  of  fundamentals . 

b.  Don't  use  as  a  required  operational  characteristic  a 
statement  such  as  "The  system  must  perform  better  than  its 

predecessor, . "  Whether  or  not  System  X  is  "better  than" 

System  Y  requires  a  judgement  that  may  (and  probably  will) 
differ  from  one  person  to  another.  If  one  sonar  has  a 
longer  detection  range  than  another,  but  provides  a  less 


accurate  bearing  to  the  target — which  one  is  better?  Depends 
on  what  you're  using  it  for.  This  is  not  to  say  that  we 
cannot  use  one  system's  performance  for  criteria  for  another. 
We  can — so  long  as  we  specify  the  parameters  we're  talking 

about.  Thus  "The  system  must  detect .  at  ranges  beyond 

the  capability  of . (X  meters)"  is  OK.  No  judgement 

required — only  analysis  of  data. 

c.  For  a  more  detailed  discussion  on  this  subject,  see 
Section  9  of  this  Guide. 

805.  Checklists .  The  attached  sheets  are  designed  to  help 
the  OTD  avoid  some  of  the  more  frequent  errors  in  COMOPTEV- 
FOR  TEMP  inputs. 
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PART  I  —  Description 
1 .  Mission 

The  operational  mission  description  really  addresses 
missxon(s),  and  not  capabilities. 

The  operational  environment  that  is  described  is 
consistent  with  the  threat. 

2a.  Key  Functions 

The  list  of  primary  functional  capabilities  is  complete. 

Primary  functional  capabilities  will  be  demonstrated 
adequately  in  DT&E  and/or  OT&E. 

2b.  Interfaces 

The  list  is  complete. 


2c.  Unique  Characteristics 


Survivability/vulnerability  aspects  are  adequately 
addressed.  ’  1 


tired  Operational  Characteristics 


They  are  operational  system  characteristics — they 
describe  (and  quantify)  the  fundamental  things  we 
require  of  the  system-and  they  are  not  technical 
characteristics .  - 


They  are  categorized  properly  into  operational 
effectiveness  and  operational  suitability. 

They  describe  things  we  can  get  a  handle  on  in  OT&E. 

They  use  numbers  as  much  as  possible,  (if  you  have  to 
suggest  a  number,  because  CNO  has  not  provided  one,  get 
with  your  Project  Analyst  and  work  out  what  you  think 
it  ought  to  be.)  - 


The  numbers  do  not  have  confidence  limits  associated  with 
ha  Acquired  operational  characteristic  is  a  mission 
reliability  of  X;  that's  what  the  system  should  have. 

limits  apply  to  our  estimates  of  what  the  system 
will  do,  not  to  what  it's  supposed  to  have.) 
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4.  Required  Technical  Characteristics 

They  are  complete  and  sufficient  to  demonstrate  the  re¬ 
quirements  of  OPNAVINST  3960.10  for  DT&E. 

They  are  sufficiently  defined  so  as  to  allow  only  one 
interpretation  of  test  results. 

5 .  Critical  T&E  Issues 

All  key  areas  of  risk  that  can  be  resolved  in  T&E  are 
addressed. 

All  discussions  are  pertinent  to  the  issues. 

PART  II  —  Program  Summary 
1 .  Management 

Responsibilities  of  participating  agencies  are  clearly 
and  properly  defined. 

Discussions  are  pertinent  and  deal  in  specifics  rather 
than  in  generalities. 

2 •  Integrated  Schedule 

All  phases  of  OT&E  are  shown. 

Phases  reflect  actual  test  operations,  not  periods 
when  we're  not  doing  anything. 

Program  structure  is  proper  (see  Section  6). 

Each  phase  of  OT&E  has  a  scheduled  Test  Plan  and  Evaluation 
Report,  and  (when  applicable)  a  Tactics  Guide. 

when  Quick-look  Reports  are  required,  they  are  scheduled 
for  release  no  later  than  14  working  days  after  completion 
of  project  operations. 

Formal  Evaluation  Reports  are  scheduled  for  publication  no 
later  than  90  calendar  days  (60  calendar  days  is  desired) 
after  completion  of  project  operations.  (Note:  in  special 
cases  involving  extensive,  time-consuming  analysis  phases, 
later  publication  dates  may  be  specified.) 

Test  articles  will  support  T&E. 
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V;  ASU  is  scheduled,  PASU  is  not.  (Do  not  preplan  PASU-- 
^  if  OT&E  is  properly  laid  out,  PASU  is  rarely  required, 
and  then  only  because  of  circumstances  that  could  not 
be  preplanned.)  (  ) 

PART  III  —  DT&E  Outline 

All  phases  of  DT&E  are  shown.  (  ) 

The  DT&E  plan  of  action  is  sufficient  to  demonstrate  the 
required  technical  characteristics  and  to  support  OT&E 
(including  S/V).  (  ) 

For  systems  with  embedded  software,  provision  is  made 

for  compliance  with  TADSTAND  9.  (  ) 

PART  IV  —  OT&E  Outline 

Each  phase  of  OT&E  has  a  defined  purpose  identified  in 

the  first  sentence  of  OT&E  Objectives — "Successful 

completion  of  OT-X  will . "  (  ) 

OT&E  Objectives  repeat  or  make  reference  to  Required 
Operational  Characteristics,  as  appropriate  to  the 
individual  phase.  (  ) 

OT&E  Events,  etc.,  describes  what  will  be  done  during 
testing — operationally,  scenario-oriented,  keyed  to  the 
threat .  (  ) 

OT-I  will  support  a  full-scale  development  decision.  (  ) 

OT-II  (OPEVAL)  will  support  an  ASU  decision.  (  ) 

Projected  OT-I I I  and  OT-IV  are  included.  (  ) 

Things  critical  to  successful  OT&E  (e.g.,  availability 

of  a  new  threat  simulator)  are  shown  as  Critical  Items.  (  ) 

PART  vi  --  Special  Resource  Summary 

Target  requirements  are  included  and  have  been  defined 
in  light  of  accurate  threat  information.  (  ) 

Fleet  RDT&E  Support  is  specified  in  detail  (hours,  etc.) 

and  Code  23  is  aware.  (  ) 

Special  instrumentation/simulations  are  identified  and 

are  being  procured/provided.  (  ) 

Requirements  for  data  processing,  analysis,  etc.  are 
;  specified.  (  ) 

8 -6a  Change  (1) 
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806.  Resource  Schedulin' 


a.  RDT&E  support  includes  operating  and  nonoperating  support  pro¬ 
vided  by  operational  naval  forces  having  a  primary  mission  other  than 
R&D;  to  the  DA,  COMOPTEVFOR,  PRESINSURV,  or  an  R&D  Agency;  for  the 
accomplishment  of  acquisition  program  T&E,  or  research  and  develop¬ 
ment  not  related  to  specific  acquisition  programs. 

b.  Theoretically,  RDT&E  support  requirements  are  compiled  from 
two  inputs : 

(1)  Approved  TEMPs  for  ACAT-I,  II,  and  III  programs  (Note: 
by  definition,  ACAT-IV  programs  do  not  require  RDT&E  support). 

(2)  Requests  for  RDT&E  support  for  R&D  not  related  to  specific 
acquisition  programs,  submitted  to  CNO  by  the  R&D  agency.  (See 
OPNAVINST  3960.10,  enclosure  (4)). 

In  actuality,  TEMPs  themselves  are  not  yet  being  used  for  this  purpose. 
So  prepare  a  separate  letter,  restating  TEMP  requirements.  Your 
counterpart  from  the  23-shop  will  help  you  with  this  —  get  with 
him/her  early  on,  and  coordinate  with  him/her  frequently. 

c.  From  these  inputs  OP-098  publishes  annually  "CNO  Long-Range 
RDT&E  Support  Requirements"  for  the  budget-  and  out-years.  Fleet 
commanders  and  others  use  this  report  for  guidance  in  planning,  pro¬ 
gramming,  and  budgeting  for  RDT&E  Support. 

d.  Using  these  same  inputs,  updated  by  confirmation  procedures, 
OP-098  publishes  quarterly  "CNO  Quarterly  RDT&E  Support  Requirements" 
for  the  forthcoming  quarter.  This  is  used  at  quarterly  fleet  schedul¬ 
ing  conferences  to  establish  requirements  for  RDT&E  Support. 

e.  OP-098  assigns  a  priority  (applying  to  fleet  support  only)  to 
each  task  in  the  CNO  Quarterly  RDT&E  Support  Requirements.  These 
priorities  are  discussed  in  OPNAVINST  3960.10. 
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Section  9 


How  to  Design  a  Phase  of  OT&E 
901.  Introduction 


a.  While  this  section  is  designed  to  be  as  self- 
sufficient  as  possible,  some  aspects  of  it  may  become 
clearer  if  you  read  Sections  10  and  11  —  all  three  sections 
are  closely  related. 


b.  There  are  many  ways  to  design  a  phase  of  OT&E  (OT- 
I IB,  OT-IIIA,  etc.).  Regardless  of  the  approach  taken,  it 
should  include  the  following  essential  elements: 

(1)  Determine  the  reason  for  the  phase. 

(2)  Decide  upon  the  objectives  and  the  evaluation 
criteria. 

(3)  Decide  if  testing  will  be  scenario-oriented  or 
operation-oriented . 

(4)  Specify  the  E-tests  (effectiveness  tests)  and 
S-tests  (suitability  tests). 


(5)  Determine  the  dcta  requirements,  and  how  they 
will  be  met. 


(6) 


how  long  the 


Decide  how  many  times  scenarios  must  be  run, 
equipment  must  be  operated. 


or 


(7)  Determine  the  resource  requirements. 

(8)  Determine  conditions  under  which  project 
operations  would  be  terminated  early. 


902 .  The  Reason  for  a  Phase  of  OT&E 


a.  There  is  a  reason  for  each  phase  of  OT&E  —  the 
reason  is  usually  associated  with  a  program-level  decision 
regarding  the  system  being  tested.  If  there  is  a  properly 
prepared  TEMP,  the  reason  for  each  phase  of  future  OT&E  will 
be  stated  in  the  first  sentence  of  the  appropriate  "OT&E 
Objectives"  paragraph  of  Part  IV  —  e.g.,  "Successful  completion 
of  OT-I  will  allow  a  recommendation  on  full-scale  development." 
Note  that  the  phase's  reason  determines  the  subject  of  the 
major  recommendation  of  the  phase's  Evaluation  Report  — 
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e.g.,  "Proceed  into  full-scale  development  of  the  ..."  If 
there  is  not  a  properly  prepared  TEMP,  then  you,  the  OTD, 
must  determine  the  reason  —  if  you  have  trouble  making  this 
determination,  chances  are  the  phase  is  unnecessary.  The 
reasons  most  frequently  associated  with  phases  of  OT&E  are 
as  follows: 

(1)  OT-O .  To  allow  a  recommendation  on  advanced 
development. 

(2)  OT-IA.  To  allow  a  recommendation  between 
competing  approaches. 

(3)  OT-I .  To  allow  a  recommendation  on  full-scale 
development. 

(4)  OT-I IA.  To  allow  a  recommendation  on  prototyping 
for  TECHEVAL/OPEVAL . 

(5)  OT- I I ( OPEVAL ) .  To  allow  a  recommendation  on 
ASU  and  production. 

(6)  OT-I II.  To  allow  a  recommendation  on  ASU  and 
full-rate  production. 

(7)  OT-IV.  To  determine  the  need  for  modification. 

b.  A  common  mistake  in  planning  phases  of  OT&E  is  to 
assign  an  OT  phase  to  every  DT  phase  —  automatically, 
without  thinking  about  it.  The  fact  that  the  DA  finds  it 
necessary  to  run  DT-IIA  before  DT-IIB(TECHEVAL)  does  not 
automatically  require  OT-IIA  before  OT-I IB (OPEVAL) .  If  a 
program-level  decision  is  to  be  made  after  DT-IIA,  then  we 
should  schedule  an  OT-IIA  concurrent  with  or  after  DT-IIA 
so  that  COMOPTEVFOR  is  on  record  as  having  an  input  to  the 
decision.  If,  on  the  other  hand,  DT-IIA  is  scheduled 
solely  to  provide  necessary  technical  data  to  the  DA,  there 
is  no  reason  for  the  OT-IIA.  Remember  two  things: 

(1)  We  can  always  observe  DT  testing  if  we  want  to, 
and  if  we  elect  to  write  a  report  to  CNO  based  on  our 
observation,  we're  free  to  do  so. 

(2)  Each  phase  of  OT&E  that  you  identify  as  such 
requires  that  you  write  a  Test  Plan  and  an  Evaluation  Report. 


903.  Objectives  and  Evaluation  Criteria 

a.  The  lead-in  paragraph  of  a  typical  OT-I  Evaluation 

Report  says  "The  purpose  of  the  evaluation  was  to  assess  the 
potential  operational  effectiveness  and  operational  suitability 
of  the  and  its  readiness  for  full-scale  development." 

A  typical  OPEVAL  report  reads  "The  purpose  of  the  evaluation 
was  to  determine  the  operational  effectiveness  and  operational 
suitability  of  the  . . . ,  and  its  readiness  for  approval  for 
service  use  and  production."  Both  these  statements  address 
the  reason  for  the  phase  being  reported  and  associate  with 
it  investigations  of  operational  effectiveness  and  operational 
suitability.  This  is  because  COMOPTEVFOR 1 s  investigation  of 
operational  effectiveness  and  operational  suitability  is  the 
basis  for  his  evaluation  —  in  the  Evaluation  Report  he 
arrives  at  conclusions  about  operational  effectiveness  and 
operational  suitability  in  order  to  recommend  action  regarding 
the  pending  program-level  decision  ( i . e . ,  the  reason) . 

Thus  each  phase  of  OT&E  is  basically  an  investigation  of 
operational  effectiveness  and  operational  suitability 
(actual  or  potential). 

b.  According  to  DOD  Directive  5000.3,  operational  effec¬ 
tiveness  is  "the  overall  degree  of  mission  accomplishment  of 
a  system  used  by  representative  personnel  in  the  context  of 
the  organization,  doctrine,  tactics,  threat  (including  coun¬ 
termeasures  and  nuclear  threats)  and  environment  in  the 
planned  operational  employment  of  the  system."  (Note  that 
this  statement  implies  that  survivability/vulnerability  is 

an  integral  part  of  operational  effectiveness.)  The  essen¬ 
tial  elements  of  operational  effectiveness  —  the  things  the 
system  must  do  (and  must  not  do)  in  order  for  mission  accom¬ 
plishment  —  vary  from  one  system  to  the  next.  Some  typical 
examples  of  elements  of  operational  effectiveness  are  pro¬ 
vided  in  Table  9-1. 
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Elements  of  Operational  Effectiveness 
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c.  According  to  DOD  Directive  5000.3,  operational  suit-  (R 
ability  is  "the  degree  to  which  a  system  can  be  satisfac¬ 
torily  placed  in  field  use,  with  consideration  being  given 
availability,  compatibility,  transportability,  interopera¬ 
bility,  reliability,  wartime  usage  rates,  maintainability, 
safety,  human  factors,  manpower  supportability,  logistic 
supportability ,  and  training  requirements." 

d.  For  a  given  system,  the  essential  elements  of 
operational  effectiveness  and  operational  suitability  form 
the  framework  for  the  objectives  of  OT&E.  That  is,  the 
objectives  define  operational  effectiveness  and  operational 
suitability  for  a  given  phase  of  OT&E.  For  example,  the 
objectives  for  OPEVAL  of  a  surface  ship  sonar,  derived  from 
Table  9-1  and  the  definition  in  the  previous  paragraph, 
would  be: 

(1)  To  determine  the  sonar's  capability  to  detect, 

classify,  and  track .  in  the  natural  acoustic  environment. 

(2)  To  determine  the  sonar's  capability  to  detect, 

classify,  and  track .  in  the  presence  of  acoustic  counter¬ 

measures  . 

(3)  To  determine  the  sonar's  capability  to  localize 

targets . 

(4)  To  determine  the  sonar's  false  alarm  rate. 

(5)  To  determine  the  sonar's  reliability,  maintain¬ 
ability,  and  availability  in  the  shipboard  environment. 

(6)  To  determine  the  sonar's  logistics  support- 
ability  in  a  deployed  status. 

(7)  To  determine  the  sonar's  compatibility  with  all 
elements  of  the  operational  environment. 

(8)  To  determine  the  sonar's  interoperability  with 
the  Underwater  Fire  Control  System,  and  the  adequacy  of  the 
sonar/operator  interfaces  (displays,  controls,  etc.). 

(9)  To  determine  the  adequacy  of  training  planned 
for  sonar  operators  and  maintenance  personnel. 

e.  These  illustrative  objectives  deserve  some  explanatory  (R 
comments . 
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(1)  They’re  all  ”to  determine.”  In  an  OPEVAL  that's 
not  unusual.  In  earlier  OT&E  when  the  equipment  doesn’t 
closely  approximate  the  planned  production  configuration 
(e.g.,  in  OT-I  with  an  advanced  development  model),  our 
objectives  couldn't  be  "to  determine"  in  all  cases.  For 
example,  we  would  probably  "estimate  the  potential  reliabil¬ 
ity,  "  and  would  not  mention  interoperability  with  the  Under¬ 
water  Fire  Control  System  if  that  interface  had  not  been 
mechanized. 

(2)  There  is  no  mention  of  transportability.  Be¬ 
cause  the  sonar  is  permanently  installed,  the  OTD  did  not 
consider  this  element  essential. 

(3)  There  is  no  mention  of  wartime  usage  rates  be¬ 
cause  the  system  is  not  expendable. 

(4)  There  is  no  mention  of  safety.  The  OTD  decided 
that  safety  deficiencies,  if  they  exist,  could  be  addressed 
under  other  objectives  (primarily  under  maintainability  and 
interoperability).  (This  is  discussed  more  fully  in  a  later 
paragraph. ) 

(5)  There  is  no  mention  of  human  factors  or  manpower 
supportability.  As  with  safety,  the  OTD  choBe  to  consider 
these  under  maintainability  and  interoperability. 

f.  None  of  the  elements  of  operational  effectiveness 
and  operational  suitability,  nor  any  of  the  illustrative 
objectives,  is  quantified  —  a  very  common  situation. 
Obviously  —  from  an  evaluation  viewpoint  —  it  is  not 
sufficient  to  say  that  the  sonar  must  detect,  and  that  it 
must  have  reliability.  It  is  also  necessary  to  say  things 
about 


(1)  What  probabilities  of  detection  are  acceptable. 

(2)  The  ranges  at  which  detection  should  occur. 

(3)  How  reliability  is  measured  (e.g.,  mission 
reliability  versus  mean  time  between  failures),  and  what  is 
acceptable . 

(4)  And  so  forth. 

These  statements  are  the  evaluation  criteria  --  their  pri¬ 
mary  function  is  to  quantify  objectives  that  need  it. 
(Occasionally  an  evaluation  criterion  won't  quantify  —  it 
will  specify  —  e.g.,  "The  pod  must  be  compatible  with  A-7, 
F-4,  and  F-14  aircraft.") 


Change  (1) 
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g.  The  objectives  of  a  phase  of  OT&E  and  the  associated 
evaluation  criteria  are  documented  in  the  TEMP  —  in  the 
appropriate  "OT&E  Objectives"  paragraph  of  Part  IV,  which 
will  usually  make  reference  to  paragraph  3  of  Part  I, 
"Required  Operational  Characteristics."  When  you  are  work¬ 
ing  from  a  TEMP  and  are  designing  a  phase  of  OT&E,  view  the 
TEMP  critically  —  don't  assume  that  the  OT&E  objectives  are 
complete  —  examine  them  carefully  to  make  sure.  If  they're 
not  complete,  make  them  so  in  your  Test  Plan  —  the  TEMP  can 
be  fixed  later.  (But  be  sure  the  DA  and  OPNAV  Sponsor  are 
aware  of  changes  in  objectives,  so  differences  of  opinion 
can  be  resolved  before  —  not  after  —  testing.)  If  you're 
not  working  from  a  TEMP,  then  you  must  define  the  objectives 
and  evaluation  criteria  from  scratch.  Again,  make  sure  the 
DA  and  OPNAV  Sponsor  get  an  early  chance  to  comment  on  them. 
Probably  the  most  critical  task  you  have  in  planning  a  phase 
of  OT&E  is  to  make  sure  that  the  objectives  and  associated 
evaluation  criteria  define  and  quantify  the  elements  of  oper¬ 
ational  effectiveness  and  operational  suitability  essential 
to  the  phase. 

904.  Scenario-Oriented  or  Operation-Oriented  Testing.  Hav¬ 
ing  determined  that  there  is  a  valid  reason  for  a  phase  of 
OT&E,  and  having  defined  and  quantified  the  elements  of 
operational  effectiveness  and  operational  suitability  that 
are  essential  to  the  phase  —  in  terms  of  objectives  and 
evaluation  criteria  —  you  are  ready  to  decide  how  the 
objectives  will  be  met  —  how  the  equipment  will  be  tested. 
The  two  methods  most  common  in  OT&E  are  scenario-oriented 
testing  and  operation-oriented  testing. 

a.  Scenario-oriented  testing  is  commonly  used  for  sys¬ 
tems  whose  modes  of  operation  or  functions  change  according 
to  a  changing  operational  situation.  For  example,  a  ship¬ 
board  antiair  fire  control  system  that  is  mostly  in  search 
mode  until  an  attack  is  a  prime  candidate  for  scenario- 
oriented  testing.  So  is  an  OBA  (Oxygen  Breathing  Apparatus) 
for  damage  control  personnel  that  is  only  used  in  emergen¬ 
cies  (barring  training  exercises  and  the  like).  In  scenario- 
oriented  testing,  the  system  under  evaluation  is  introduced 
into  a  realistic  simulation  of  a  developing  operational 
situation  --  a  scenario  --  and  is  put  through  its  paces 
pursuant  to  its  mission/function  while  being  observed  by 
data  recorders  (human  and  machine). 

(1)  For  a  shipboard  fire  control  system,  a  scenario 
could  simulate  open-ocean  transit  of  a  Task  Force.  The  test 
ship  with  the  fire  control  system  is  assigned  to  search  a 
sector  and  engage  penetrating  air  targets.  At  a  preplanned 
time  not  announced  to  ship's  company,  a  raid  consisting  of 
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electronic  warfare  and  strike  aircraft  attempts  to  penetrate 
the  test  ship's  sector.  Some  strike  aircraft  target  the 
test  ship,  others  target  other  elements  of  the  Task  Force. 

The  test  ship  acquires  the  penetrating  aircraft,  prioritizes 
them,  and  engages  them  —  kills  are  "called"  by  the  OTD. 

(2)  For  the  OBA,  a  scenario  could  simulate  damage 
(with  fire)  in  a  compartment  distant  from  the  damage  control 
personnel.  Their  job  would  be  to  don  their  gear;  pick  up 
their  toois,  firefighting  equipment,  etc.;  proceed  to  the 
scene  of  the  damage;  extinguish  the  fire  and  perform  neces¬ 
sary  damage  control  tasks;  and  stow  their  gear. 

(3)  Multipurpose  systems  may  require  several 
scenarios  to  exercise  their  various  capabilities. 

The  data  recorded  during  the  scenarios  are  used  for  recon¬ 
struction  and  analysis  in  the  various  E-tests  and  S-tests 
discussed  below.  Often,  scenario-oriented  testing  is  dedi¬ 
cated  testing  (in  terms  of  fleet  RDT&E  support)  —  although 
it  can  be  accomplished  on  a  not-to-interfere  basis  during 
fleet  exercises. 

b.  Operation-oriented  testing  is  commonly  used  for 
equipment  whose  mode  of  operation  or  function  remains  con¬ 
stant.  Torpedo  tubes,  communications  receivers,  and  sewage 
disposal  systems  are  essentially  either  "in  use"  or  "not  in 
use"  and  can  be  tested  by  just  operating  them  —  making  sure, 
of  course,  that  the  operating  conditions  reflect  the  antici¬ 
pated  environment.  The  latter  may  require  scheduling  of 
services  —  targets,  jammers,  etc.  —  otherwise,  operation- 
oriented  testing  can  frequently  be  accomplished  on  a  not-to- 
interfere  basis. 

c.  If  you,  the  OTD,  have  difficulty  deciding  between 
scenario-oriented  and  operation-oriented  testing  for  a  phase 
of  OT&E,  then  choose  scenario-oriented  testing.  That  way 
you'll  have  some  assurance  that  test  results  are  reasonable 
indicators  of  performance  that  can  be  expected  in  the  fleet  — 
and  there  will  be  a  greater  chance  that  the  unexpected  will 
happen.  (See  paragraph  302. e  for  more  words  on  this  subject. ) 

d.  If  you  have  decided  on  scenario-oriented  testing, 
your  next  task  is  to  design  the  scenario(s)  —  to  describe 
the  exercise(s)  that  will  stress  the  system  in  a  realistic 
manner.  Describe  the  tactical  situation  at  the  start  of  the 
exercise  (e.g.,  open-ocean  steaming  with  a  Task  Force  expect¬ 
ing  to  be  engaged  by  the  enemy).  Then  describe  the  situation 
that  develops  (incoming  aircraft  spotted,  etc.)  and  the 
actions  required  of  the  system  under  test  (e.g.,  test  system 
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is  to  acquire  and  engage  attacking  aircraft).  When  you  are 
finished,  you  should  have  a  narrative  that  describes  the 
exercise,  start  to  finish;  supplement  this  narrative  as 
necessary  with  diagrams  or  tables  specifying  the  movements 
of  exercise  participants  as  a  function  of  time,  and  their 
actions  (e.g.,  aircraft  commence  jamming  at...).  This 
narrative,  supplemented  as  necessary,  is  a  scenario;  make  as 
many  as  the  system  demands,  in  your  judgement. 

e.  If  you  have  decided  on  operation-oriented  testing, 
your  next  task  is  to  specify  the  events  and  conditions 
necessary  during  system  operation  —  for  instance  the  tar¬ 
gets  ,  the  environments  ( natural  and  man-made ) ,  etc .  Hake 
sure  your  events  and  conditions  provide  an  operationally 
realistic  test  of % the  system. 

f.  A  properly  prepared  TEMP  will  already  have  specified 
the  type  of  testing,  and  will  have  provided  a  skeletal  out¬ 
line  of  it  in  the  appropriate  "OT&E  Events/Scope  of  Testing/ 
Basic  Scenarios"  paragraph  in  Part  IV. 

g.  When  you  are  designing  a  scenario,  or  when  you  are 
making  plans  for  equipment  operation,  keep  the  following  in 
mind: 


(1)  Testing  should  involve  simulations  of  enemy 
counteractions  —  maneuvers  he  might  make,  electronic  war¬ 
fare  techniques  he  might  employ,  etc.  —  in  order  that  the 
system's  vulnerability  to  these  actions  may  be  assessed. 

For  example,  in  tests  of  a  surface  ship  sonar,  include  a 
target  submarine  employing  acoustic  countermeasures.  (This 
does  not  mean  that  all  the  sonar  tests  should  include  acous¬ 
tic  countermeasures;  during  the  system's  projected  opera¬ 
tional  life,  the  most  frequent  targets  may  be  submarines  not 
employing  countermeasures.  Therefore,  operational  testing 
should  employ  both  types  of  targets.)  See  Annex  D  for  spe¬ 
cifics  on  how  to  get  quidance  on  enemy  capabilities  and 
tactics . 


(2)  The  environment  in  which  the  system  is  tested 
should  approximate,  as  closely  as  possible,  the  anticipated 
operational  environment.  As  necessary,  depending  on  the 
type  of  system  being  tested,  you  should  provide  for: 

(a)  The  anticipated  "noise"  background  caused  by 
other  ships,  aircraft,  etc.  in  the  area,  to  allow  evaluation 
of  effects  such  as  EMI . 

(b)  Operation  of  other  equipment  that  might  be 
expected  to  be  used  simultaneously  with  the  system  under 
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test,  to  allow  evaluation  of  effects  of  changes  in  electrical 
power  loads,  effects  of  gunfire-induced  shock  and  vibration, 
etc. 

905.  Specifying  the  E-Tests  and  S-Tests 


a.  E-tests  and  s-tests  are  usually  conducted  at  your 
desk,  after  scenarios  have  been  run  or  operation-oriented 
testing  is  complete.  During  E-  and  S-tests,  data  are  ana¬ 
lyzed  to  find  out  how  the  test  system  performed  during  the 
scenario (s)/operation,  what  its  reliability  was,  etc.  Each 
E-  and  S-test  addresses  an  objective  of  the  phase  of  OT&E, 
an  aspect  (piece)  of  an  objective,  or  an  aspect  of  several 
objectives.  Their  function  is  to  determine  the  things  we 
need  to  know  about  the  system  —  quantitative  things  (the 
various  MOEs  (measures  of  effectiveness)  and  MOSs  (measures 
of  suitability))  and  qualitative  things  (the  adequacy  of 
technical  manuals,  the  Integrated  Logistics  Support  Plan, 
the  Navy  Training  Plan,  etc.). 

b.  To  determine  what  E- tests  are  necessary,  the  OTD 
must  examine  each  operational  effectiveness  objective  and 
decide  what  needs  to  be  known  to  meet  each  objective.  For 
example,  consider  the  first  objective  of  paragraph  903. d  — 

Mto  determine  the  sonar’s  capability  to  detect,  classify, 
and  track...  in  the  naturar  acoustic  environment."  What 
does  the  evaluator  need  to  know  to  meet  this  objective?  The 
following  come  to  mind: 

(1)  How  often  does  detection  occur  against  targets 
that  should  be  detected?  (The  conditions  that  define  "should 
be  detected"  should  have  been  specified  in  the  evaluation 
criteria. ) 

(2)  At  what  ranges  does  detection  occur?  (Operation¬ 
ally  useful  ranges  must  be  defined  in  the  evaluation  criteria.) 


occur? 


(3)  Given  detection,  how  often  does  classification 


(4)  Of  the  classifications,  how  many  are  correct? 

(5)  Of  the  incorrect  classifications,  how  many  are 
critical  (i.e.,  threat  classified  as  non-threat)? 


occur? 


(6)  How  long  after  detection  does  classification 


(7)  At  what  ranges  do  classifications  occur? 
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(8)  Given  detection,  how  often  can  a  track  be  estab¬ 
lished  on  targets  that  should  be  tracked?  ("Should  be 
tracked"  conditions  should  be  specified  in  the  evaluation 
criteria.) 

(9)  How  long  after  detection  are  tracks  established? 

(10)  At  what  ranges  are  tracks  established? 

(11)  Given  established  tracks,  how  well  are  tracks 
held  that  should  be  held?  ("Should  be  held"  conditions 
should  be  specified  in  the  evaluation  criteria.) 

These  11  questions  suggest  the  following  E-tests  and  associated 
NOEs : 

(1)  Test  E-l,  Detection. 

(a)  MOE  1  —  Probability  of  detection. 

(b)  MOE  2  —  Detection  range. 

(2)  Test  E-2 ,  Classification. 

(a)  MOE  3  —  Probability  of  correct  classifica¬ 
tion,  given  detection. 

(b)  MOE  4  —  Probability  of  classifying  a 
threat  as  a  non- threat. 

(c)  MOE  5  —  Time  between  detection  and  classifica¬ 
tion. 

(d)  MOE  6  —  Classification  range. 

(3)  Test  E-3,  Tracking. 

(a)  MOE  7  —  Probability  of  establishing  a 
track,  given  detection. 

(b)  MOE  8  —  Time  between  detection  and  track 
establishment. 

(c)  MOE  9  —  Range  at  track  establishment. 

(d)  MOE  10  —  Percent  of  time  tracks  are  held. 

(Note  that  in  this  example ,  you  need  to  know  quantita¬ 
tive  things  in  order  to  meet  the  objective  —  things 

that  can  be  expressed  as  MOEs.  There  will  often  be 
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qualitative  things  you  must  know  in  addition  to  the 
MOEs. ) 


Having  defined  the  E- tests  and  MOEs  for  the  first  objective, 
the  OTD  examines  the  second  —  "to  determine  the  sonar's 
capability  to  detect,  classify,  and  track  ....  in  the  pre¬ 
sence  of  acoustic  countermeasures."  He  notes  that  it  is  the 
same  as  the  first  objective  —  except  that  acoustic  counter¬ 
measures  have  been  added  —  and  elects  to  treat  the  acoustic 
environment  as  a  variable  is  Tests  E-l  through  E-3.  That  is, 
he  decides  to  calculate  MOEs  1  through  10  twice  --  with  and 
without  acoustic  countermeasures.  Had  he  desired  to  do  so, 
he  might  have  specified  (for  example)  a  Test  E-l,  Detection 
(Natural  Environment),  a  Test  E-2,  Detection  (Countermeasures), 
and  so  on.  After  having  taken  care  of  the  second  objective, 
the  OTD  proceeds  to  the  remaining  two  effectiveness  objectives. 

c.  The  process  of  selecting  S-tests  consists  first  of 
chosing  the  applicable  tests  from  the  list  of  standardized 
suitability  tests  (in  paragraph  1002),  and  then  adding 
others  as  necessary.  Which  tests  are  selected  will  vary 
according  to  the  system  under  test,  and  the  phase  of  OT&E. 

The  following  general  guidelines  apply: 

(1)  Reliability.  A  test  of  reliability  is  appropriate 
when  the  test  system's  design,  construction,  and  installation 
approximate  those  of  the  proposed  production  system  --  e.g., 

in  OPEVAL  and  OT-III.  In  these  phases  of  OT&E,  it  is  possible 
to  estimate  the  reliability  of  the  operational  system  based 
on  performance  of  the  test  system.  In  earlier  phases  of 
OT&E,  when  the  test  system  is  functionally  equivalent  to  the 
production  system,  but  is  much  different  physically  (for 
example,  a  brassboard),  extrapolation  of  MTBFs,  etc.,  to  the 
production  configuration  is  not  possible.  In  some  systems 
it  is  possible,  even  early  in  the  design  phase,  to  identify 
potential  reliability  problem  areas  —  based,  for  example, 
on  the  system' s  use  of  components  known  to  have  high  failure 
rates  in  similar  equipment.  Whether  or  not  to  include  a 
reliability  test  in  early  IOT&E  is  a  matter  of  judgement. 

(2)  Maintainability.  The  conditions  under  which  a 
maintainability  test  is  appropriate  are  very  similar  to 
those  for  a  reliability  test.  Keep  in  mind,  however,  that 
maintainability  parameters  such  as  mean  times  to  fault- 
locate  and  to  repair  have  little  meaning  from  an  operational 
viewpoint  unless  maintenance  is  accomplished  by  fleet-type 
personnel  —  whereas  this  is  not  necessarily  the  case  for 
reliability  parameters.  In  addition,  there  are  occasions 
when  maintainability  is  not  an  issue.  For  example,  a  target 
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drone  that  is  maintained  under  a  maintenance  agreement  (con¬ 
tract)  has  reliability  and  availability  parameters  in  it6 
required  operational  characteristics,  but  not  maintainabil¬ 
ity  parameters. 

(3)  Availability.  Usually,  when  mission  reliability 
is  an  issue,  operational  availability  (the  probability  the 
system  will  be  ready  to  begin  its  mission)  is  also. 

(4)  Logistics  Supportability.  This  test  is  usually 
required  in  OPEVAL  and  OT-III.  Some  systems  that  are  produc¬ 
tion-prototyped  early  —  systems  used  in  explosive  ordnance 
disposal  frequently  are  —  can  be  examined  from  a  logistics 
supportability  viewpoint  earlier  in  IOT&E.  Systems  that 
have  unusual  servicing  requirements  (e.g.,  pressurizing  with 
an  uncommon  gas)  or  that  use  short-lived  or  extremely  deli¬ 
cate  parts  should  also  be  examined  early,  to  identify  poten¬ 
tial  support  problems  in  the  fleet. 

(5)  Compatibility.  This  test  is  also  usually 
required  in  OPEVAL  and  OT-III.  Furthermore,  even  though  the 
test  system  is  an  advanced  development  model  in  a  temporary 
installation,  compatibility  tests  during  early  IOT&E  may 
reveal  problems  not  anticipated  by  the  designer  —  need  for 
an  air  conditioned  space,  a  susceptibility  to  degradation 
from  input  power  variations,  an  unanticipated  EMI  source, 
etc.  Early  identification  of  potential  compatibility  prob¬ 
lems  may  allow  simple  changes  (e.g.,  installation  in  a  dif¬ 
ferent  location)  that  later  prevent  the  system  from  failing 
in  OPEVAL. 

(6)  Interoperability.  Checks  on  the  man/machine 
interface  usually  begin  in  the  first  phase  of  IOT&E.  Inter¬ 
faces  between  the  test  system  and  associated  systems  are 
tested  whenever  they  have  been  mechanized.  Interoperability 
testing  usually  continues  through  OT-III. 

(7)  Training  Requirements.  This  test  is  conducted 
as  soon  as  a  proposed  training  plan  has  been  defined,  and  is 
repeated  as  necessary  through  OPEVAL. 

(8)  Transportability.  This  test  is  conducted  if  it 
is  appropriate  to  the  system  under  test,  and  when  the  con¬ 
figuration  of  the  test  item  allows  a  meaningful  test.  Items 
designed  to  be  man-portable  are  frequently  in  near-produc- 
tion  configurations  early  in  their  development,  and  trans¬ 
portability  testing  can  begin  correspondingly  early. 

(9)  Safety.  As  was  mentioned  earlier  in  this  sec¬ 
tion,  procedures  for  checking  safety  aspects  of  a  system  are 
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frequently  included  as  part  of  maintainability  and  inter¬ 
operability  tests.  When  safety  is  a  primary  reason  for 
developing  a  system  —  e.g.,  a  life  support  system  —  safety 
is  usually  addressed  in  the  system's  operational  effective¬ 
ness  objectives  (e.g.,  the  capability  of  the  system  to  sup¬ 
port  life).  The  same  is  true  of  systems  developed  to  perform 
hazardous  tasks  (e.g.,  explosive  ordnance  disposal  equipment 
—  to  determine  the  system's  capability  to  contain  the  ef¬ 
fects  of  bomb  detonation,  for  example).  Systems  not  devel¬ 
oped  for  safety  reasons  that  involve  potentially  hazardous 
operations  usually  require  a  safety  test.  For  example,  OPE- 
VAL  of  a  swimmer-delivered  remotely  controlled  limpet  mine 
should  include  a  safety  test  that  addresses  the  possibility 
of  premature  or  inadvertent  actuation. 

(10)  Human  Factors.  As  with  safety,  whether  or  not 
to  include  a  specific  human  factors  test  frequently  depends 
on  the  way  the  OTD  wants  to  structure  the  suitability  evalu¬ 
ation.  Including  human  factors  aspects  in  maintainability 
and  interoperability  tests  frequently  obviates  the  need  for 
a  specific  human  factors  test.  Specific  human  factors  tests 
are  usually  used  in  evaluations  in  which  human  factors  is  a 
major  issue  —  e.g.,  evaluation  of  a  new  flight  suit. 

(11)  Wartime  Usage  Rates.  Systems  that  contain  ele-  (R 
ments  that  will  be  expended  (e.g.,  gun  systems  (ammunition), 
missile  systems  (missiles),  countermeasures  systems  (chaff, 
expendable  decoys),  need  to  be  examined  for  assurance  that 
storage,  resupply,  etc.  facilities  will  be  adequate  in  war¬ 
time.  This  element  of  operational  suitability  is  frequently 
addressed  in  logistics  supportability. 

(12)  Manpower  Supportability.  This  test  may  be  con-  (R 
ducted  as  a  subset  of  maintainability  and  interoperability 
tests,  or  as  a  separate  test.  It  is  conducted  as  a  separate 
test  when  manning  is  a  critical  development  issue. 

d.  The  process  of  selecting  MOSs  is  the  same  as  the 
process  of  selecting  MOEs  —  determining  what  the  evaluator 
needs  to  know  to  meet  the  suitability  objectives.  Using  the 
thought  process  described  for  effectiveness  tests,  but 
recognizing  that  qualitative  things  usually  have  to  be  known 
to  meet  the  suitability  objectives,  the  OTD  will  generate 
something  like  the  following: 

(1)  Test  S— 1 ,  Reliability. 

(a)  MOS  1  —  Mean  time  between  critical/major 

failures . 
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(b)  Factors  that  appear  to  affect  reliability. 

(2)  Test  S-2,  Maintainability. 

(a)  MOS  2  ~  Mean  time  to  fault-locate. 

(b)  MOS  3  —  Mean  time  to  repair. 

(c)  Aspects  of  maintenance  that  are  excessively 
difficult,  time-consuming,  or  unsafe. 

(d)  The  adequacy  of  technical  documentation 
used  in  maintenance. 

(e)  The  adequacy  of  the  proposed  preventive 
maintenance  schedule. 

(3)  And  so  on. 

e.  Having  specified  the  tests  and  the  things  to  be 
determined  in  each,  the  OTD  can  construct  something  like 
Table  9-2.  A  table  like  this  becomes  especially  useful  in 
conq>licated  OT&E  —  for  example,  a  whole-ship  OPEVAL  — 
where  there  may  be  many  objectives  and  sub-objectives. 

Mote:  Effectiveness  and  suitability  analysts  are 
experts  at  designating  tests  and  in  selecting  MOEs, 
MOSs,  etc.  —  be  sure  to  get  them  involved  in  your 
planning  early. 
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f.  In  the  preceding  discussion,  it  was  implied  that  the 
testing  would  consist  either  of  scenario  run-throughs  or 
operation  of  the  equipment  under  simulated  operational 
conditions.  While  these  exercises  will  usually  satisfy  most 
of  the  requirements  of  the  evaluation,  additional  test 
procedures  to  be  performed  in  addition  to  the  exercises  are 
often  required.  For  example: 

(1)  Survivability /vulnerability  frequently  requires 
an  assessment  of  the  physical  installation  —  to  determine 
if  a  system  is  vulnerable  to  a  "cheap  kill."  An  example  of 
the  special  procedures  involved  is  contained  in  Section  10. 

(2)  Maintainability  frequently  requires  a  maintain¬ 
ability  demonstration  —  inserting  prefaulted  components  in 
the  equipment  and  observing  fault-location  and  repair.  In 
evaluations  where  MTFL,  MTTR,  etc.  are  issues,  always  make 
provisions  for  a  maintainability  demonstration  —  so  that 
the  maintainability  of  a  highly  reliable  system  can  be 
assessed. 


(3)  Compatibility  requires  that  equipment  not 
associated  with  the  test  system  be  energized  and  deenergized 
and  that  power  variations  be  induced  —  when  the  scenarios/ 
equipment  operation  do  not  provide  a  complete  set  of  com¬ 
patibility  data,  special  turn-on/turn-off  tests  and  the  like 
must  be  planned. 

906 .  Data  Requirements 

a.  MOEs,  MOSs,  and  qualitative  things  you  need  to  know 
to  meet  the  objectives  are  determined  during  post- test 
analysis  —  after  scenarios  have  been  ram  and  the  equipment 
under  test  has  been  secured.  This  post- test  analysis  uses 
data  recorded  during  or  shortly  after  equipment  operation  — 
in  your  planning,  you  must  decide  what  data  you  need  and  how 
they  will  be  acquired.  These  decisions  should  involve 
thoughtful  consideration  of  data  sources  that  can  be  used, 
and  what  data  are  actually  required  (including  measurements, 
with  their  inherent  degrees  of  accuracy).  These  decisions 
may  affect  earlier  elements  of  your  evolving  Test  Plan  — 
e.g.,  the  way  the  scenarios  were  to  be  run.  (Planning 
usually  involves  iteration  between  various  elements  of  the 
plan. )  To  illustrate  this  —  suppose  the  OTD  had  tentatively 
decided  on  open- ocean  freeplay  between  a  surface  ship  and  a 
submarine.  Later  he  determines  that  the  relative  positions 
of  the  two  vessels  must  be  reconstructed  with  precision  in 
order  to  determine  a  set  of  MOEs.  This  forces  him  to  use  a 
range,  and  "open-ocean  freeplay"  is  modified  accordingly. 


b.  The  major  sources  of  data  available  to  the  OTD 
include : 

(1)  The  system  under  test.  Data  are  best  obtained 
from  the  system  under  test  by  observing  displays  on  the 
system  while  it  is  in  operation  —  scopes,  meters,  indicator 
lamps,  etc.  —  and  recording  display  data  manually  or  by 
instruments  (e.g.,  cameras)  not  connected  to  the  system. 

This  requires  no  alteration  to  the  test  system  —  a  definite 
plus.  Data  sources  that  require  alterations  —  hanging 
scopes  and  meters  on  the  back  of  the  console,  etc.  —  should 
be  used  only  with  caution.  If  they  were  successfully  used 
in  earlier  DT&E  (e.g.,  during  TECHEVAL  prior  to  OPEVAL),  any 
installation  problems  —  impedance  mismatches,  ground 

loops,  etc.  —  that  may  have  affected  overall  system  performance 
have  probably  been  discovered  and  corrected.  If  they  were 
not  used  before,  use  them  in  OT&E  only  as  a  last  resort  — 
and  allow  sufficient  pre-scenario  or  pre-operation  time  for 
debugging.  External  data  sources  connected  to  the  equipment 
under  test  —  whether  used  in  earlier  DT&E  or  not  —  should 
be  examined  critically  from  the  viewpoint  of  their  effect  on 
operational  realism.  Data  sources  should  not  provide  the 
operator  with  useful  information  not  available  to  him  in  the 
proposed  production  configuration,  and  so  forth. 

(2)  Equipment  already  in  service  use.  Navigation 
systems,  radars,  sonars,  communications  systems,  etc.  available 
in  the  fleet  are  potential  data  sources  that  may,  in  fact, 
determine  the  class  or  type  of  ships  or  aircraft  to  be  used 

in  OT&E.  For  example,  absolute  position  requirements  for 
reconstruction  may  dictate  that  the  test  ship  have  an  inertial 
system  on  board;  relative  position  requirements  may  dictate 
that  a  participating  ship  have  a  certain  type  of  search 
radar.  Use  of  equipment  already  installed  in  fleet  units 
can  help  reduce  the  costs  of  OT&E,  by  reducing  the  need  for 
special  instrumentation  for  test  purposes. 

(3)  Test  support  activity/range  equipment.  Track 
plots,  bomb  impact  data,  electronic  warfare  simulator  logs, 
etc.  that  are  normally  produced  by  ranges  and  other  test 
support  activities  require  no  unusual  tasking  to  obtain 
them,  and  their  production  (per  se)  does  not  detract  from 
operational  realism. 


(4)  Special  purpose  instrumentation.  Under  this 
heading  fall  the  instruments  not  available  in  the  fleet  or 
through  test  support  activities  that  are  used  to  monitor 
elements  of  the  scenario  external  to  the  system  under  test. 

In  this  group  are  on-board  cameras  aimed  at  incoming  targets 
that  record  the  effects  of  gunfire,  and  portable  voice 
recorders  used  by  observers  of  a  simulated  combat  engagement. 


(5)  Personnel  operating  or  maintaining  the  equip¬ 
ment.  In  addition  to  recording  data  In  operating  logs  and 
mainenance  records  as  required  by  the  Test  Plan,  these 
personnel  are  sources  of  qualitative  data  —  in  question¬ 
naires  and  through  interviews.  In  this  way,  for  example, 
the  adequacy  of  technical  manuals  is  usually  determined. 

(6)  The  OTP  Journal.  See  Section  15. 

(7)  DT&E  and  fleet  data. 

(a)  COMOPTEVFOR ' s  evaluation  of  any  system 
should  be  based  on  a  review  of  all  pertinent  data,  regard¬ 
less  of  the  source.  But  the  data  must  be  pertinent  —  that 
is,  if  data  were  acquired  during  non-OT&E,  there  must  be 
every  reason  to  assume  that  the  same  data  would  have  resul¬ 
ted  from  OT&E.  In  determining  whether  or  not  data  are  per¬ 
tinent  for  operational  evaluation,  ask  the  following  ques¬ 
tions  regarding  the  conditions  under  which  the  data  were 
collected: 


1.  Who  operated  the  system?  If  contractors 
did,  most  results  are  useless  for  OT&E. 

2.  Who  maintained  the  system?  If  fleet 
sailors  operated  it,  but  contractors  maintained  it,  there 
may  be  some  useful  effectiveness  and  interoperability  data; 
reliability  and  maintainability  data  may  be  used  with  cau¬ 
tion. 


3.  What  was  the  test  environment?  Aboard 
ship,  at  sea?  Sea  state?  ECM?  In  other  words,  how  closely 
did  the  test  environment  simulate  the  operational  realism 
associated  with  OT&E?  Having  established  this,  you  may 
decide  to  use  some  data  and  disregard  some  others. 

4.  Was  the  system  altered  or  modified  in 
any  way  during  or  since  the  testing?  If  hardware  or  soft¬ 
ware  changes  were  made,  be  very  selective  in  your  use  of 
pre-change  data.  Make  sure  the  change  didn’t  nullify  ear¬ 
lier  data. 


(b)  The  two  major  potential  data  sources  outside 
OPTEVFOR  are: 


1 .  DT&E  for  I OT&E  (including  OPEVAL) . 

2.  Fleet  data  for  FOT&E. 
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During  FOT&E,  it  sometimes  happens  that  COMOPTEVFOR  is  eval¬ 
uating  systems  that  have  already  been  deployed  in  signifi¬ 
cant  numbers.  In  these  cases,  it  is  necessary  that  the  Test 
Plan  make  provisions  for  obtaining  data  on  systems  deployed 
in  non-project-ships.  Actual  fleet  experience  can  provide 
essential  information  to  an  evaluation  of  operational  effec¬ 
tiveness,  and  3-M  data  can  be  very  useful  in  expanding  the 
overall  operational  suitability  data  base. 

(A  Note  on  the  Form  in  Which  Data  Are  Obtained. 

Not  only  data  sources  vary,  but  the  form  in 
which  data  can  be  obtained  varies  also .  Thus  in 
FOT&E,  operating  times,  system  status,  and  main¬ 
tenance  information  can  be  obtained  from  special 
OT&E  forms  completed  by  operators  and  mainte¬ 
nance  personnel.  At  the  same  time,  it  may  be 
possible  to  obtain  the  same  data  from  standard 
Navy  operational  forms  that  are  already  being 
completed  on  the  system  --  standard  Equipment 
Logs  and  Maintenance  Action  Forms.  When  you  can 
obtain  OT&E  data  from  logs,  charts,  forms,  etc. 
being  completed  routinely,  do  so  because: 

1.  The  record  already  exists  -  no  special 
tasking  is  required  other  than  making  sure 
OPTEVFOR  gets  a  copy. 

2.  Recording  the  data  will  not  affect 
operational  realism,  because  recording  is  part 
of  the  operational  routine.) 

c.  Deciding  what  data  are  actually  required  is  similar 
to  deciding  what  needs  to  be  known  to  meet  each  objective 
(paragraph  905).  Consider  each  MOE,  MOS,  and  qualitative 
element  within  the  framework  of  potential  data  sources,  and 
double-check  for  impact  on  earlier  phases  of  planning  (sce¬ 
narios,  etc.).  Some  examples: 

(1)  MOE  1  —  Probability  of  Detection.  The  thing 
we’re  after  here  is  the  ratio  of  the  number  o t  detections  to 
the  number  of  targets  that  should  have  been  detected. 

Assume  the  scenario  is  being  run  on  AUTEC  and  that  AUTEC  is 
tracking  both  the  surface  ship  and  the  submarine.  Assume 
also  that  the  OTD  is  observing  the  sonar  operator  and  has 
radio  communication  with  AUTEC' s  plotting  center.  The  OTD 
relays  "DETECTION"  to  AUTEC  when  the  sonar  operator  calls 
it  --  and  records  the  operator's  initial  estimate  of  range 
and  bearing,  together  with  the  time  of  the  call.  The  required 
pieces  of  data  are: 


(a)  A  time- annotated  plot  of  the  two  tracks, 
with  ship  and  submarine  positions  marked  at  "DETECTION." 
(Provided  by  AUTEC.) 

(b)  Sonar  operator's  range  and  bearing  estimates 
at  detection  —  to  confirm  that  the  detection  was  not  a 
false  detection.  (OTD  Journal.) 

(c)  Acoustic  conditions  on  the  range,  to 
establish  the  conditions  under  which  a  submarine  "should  be 
detected."  (Provided  by  AUTEC.) 

Note:  In  the  process  described  above,  the  OTD 
considered  the  data  he  needed  and  how  they  would 
be  obtained  —  to  the  extent  of  considering 
actions  and  responsibilities  during  the  exercise 
on  AUTEC.  Test  planning  requires  that  the  OTD 
consider  both  past  events  (e.g.,  selection  of  a 
scenario)  and  future  events  (e.g.,  assigning 
responsibilities  during  project  operations)  when 
he  is  addressing  a  particular  phase  of  planning. 

(2)  MOS  1 —  Mean  Time  Between  Critical/Major 
Failures.  Here  we're  after  total  sonar  operating  time 
divided  by  the  number  of  critical  and  major  failures.  The 
required  pieces  of  data  are: 

(a)  A  chronological  record  of  system  status  — 
for  operating  time,  failure  times,  and  the  operator's  assess¬ 
ment  of  the  type  of  failures.  (From  sonar  operator's  log. 

Data  Form  S-l.) 

(b)  Confirmation  of  the  type  of  failures. 

(From  maintenance  log.  Data  Form  S-2.) 

d.  Having  determined  the  data  requirements  for  the 
various  MOEs,  etc.,  the  OTD  can  construct  something  like 
Table  9-3  (which,  for  illustrative  purposes,  is  based  on 
Table  9-2) .  Notice  that  the  title  of  Table  9-3  is  "Primary 
Data  Sources."  Back-up  data  sources  are  very  important  too; 
they  can  make  the  difference  between  objective  accomplishment 
and  non- accomplishment.  In  the  surface  ship/submarine 
exercise  on  AUTEC,  loss  of  communications  to  AUTEC  (for  the 
"/  ETECTION"  transmission)  or  loss  of  AUTEC 's  plotting  capability 
could  be  offset  by  correlating  navigation  information  from 
both  vessels  —  equipment  already  in  service  use  and  the  OTD 
Journal  could  constitute  a  back-up  data  source  for  MOE  1 
(and  others) .  Notes  expanded  and  transcribed  into  the  OTD 
Journal  could  be  a  back-up  for  a  portable  voice  recorder 
with  a  bad  battery. 


Primary  Data  Sources 


£.  One  final  consideration  in  this  phase  of  planning  -■>- 
it  is  advisable  that  you  identify  any  data  items  that,  if 
not  obtainable  during  an  exercise  in  which  they  were  supposed 
to  be  obtained,  should  cause  testing  to  be  suspended.  For 
example,  if  AUTEC  were  the  only  source  of  time-position 
information,  and  if  shortly  after  COMEX  (commence  exercise) 
AUTEC' s  plotter  crapped  out,  the  prepared  OTD  would  suspend 
the  operation  —  realizing  that  the  exercise  would  not 
contribute  any  useful  data  to  most  HOEs. 

907 .  How  Many  or  How  Long? 

a.  If  you  haven't  gotten  your  analysts  involved  yet, 
you  better  do  so  now. 

b.  Determining  how  many  times  to  run  a  scenario,  or  how 
long  to  operate  the  equipment,  is  a  matter  of  judgement  that 
involves  the  following  interrelated  and  sometimes  conflicting 
considerations : 

(1)  The  variables  that  are  involved.  If,  for 
example,  we're  interested  in  a  craft's  capability  to  deploy 
and  retrieve  UDT  personnel,  we  need  runs  that  at  least 
sample  various  combinations  of  environmental  conditions 
(day/night,  sea  state,  etc.)  in  order  to  arrive  at  an  evalua¬ 
tion. 

(2)  The  degree  of  statistical  confidence  we  want  in 
our  results.  If,  for  example,  we  want  to  be  sure  that  a 
system's  MTBF  is  at  least  300  hours  with  80%  confidence, 
then  we  must  operate  the  system  for: 

(a)  900  hours  if  it  breaks  once. 

(b)  1300  hours  if  it  breaks  twice. 

(c)  1650  hours  if  it  breaks  three  times. 


If  we  demand  90%  confidence,  we  must  operate  the  system  for: 

(a)  1160  hours  if  it  breaks  once. 

(b)  1600  hours  if  it  breaks  twice. 

(c)  2000  hours  if  it  breaks  three  times. 

(3)  The  cost  of  testing.  It  costs  money  to  expend 

weapons  and  targets,  to  operate  ships  and  aircraft,  to 
operate  a  range,  and  so  on.  This  money  has  to  be  budgeted  — 
and  is  usually  in  short  supply. 


(4)  The  availability  of  fleet  services  and  range 
support.  These  usually  boil  down  to  matters  of  priority 
among  competing  requirements. 

(5)  The  time  available.  Although  COMOPTEVFOR's 
input  is  important  in  milestone  decisions,  it  is  not  the 
only  input.  Furthermore,  budgetary  considerations  often 
require  that  decisions  be  made,  if  at  all  possible,  by 
certain  dates.  For  these  reasons,  it  is  often  desirable 
that  testing  be  conducted  so  as  to  provide  only  those  data 
absolutely  essential  to  a  COMOPTEVFOR  evaluation. 

c.  There  are,  then,  no  quantitative  rules  or  guidelines 
for  determining  how  many  or  how  long. 

d.  A  properly  prepared  TEMP  will  contain  an  estimate  of 
how  many  or  how  long. 


908.  Resource  Requirements.  This  step  is  fairly  simple. 

You  know  the  participants  for  each  scenario  (or  operation), 
and  you  know  how  many  times  (or  how  long)  you'll  be  running 
them.  Determining  total  requirements  is  a  matter  of  careful 
addition  —  and  double-checking  to  make  sure  you  haven't 
left  anything  out.  A  matrix  of  scenarios  versus  requirements 
(e.g.,  Table  9-4)  may  prove  useful  during  testing  if  an 
asset  is  cancelled  unexpectedly.  Similar  lists  have  been 
helpful  in  justifying  requirements.  Note  that  if  the  list 
identifies  requirements  that  exceed  those  planned  in  the 
7FMP  (Part  VI),  you  may  have  a  problem,  depending  on  cost, 
lead-time  requirements,  etc.  (If  you  think  about  this,  you 
will  realize  that  the  process  of  identifying  resource  require¬ 
ments  in  the  TEMP  is  a  critical  process  for  OT&E  not  far 
down  the  pike.  If  these  resource  requirements  were  hip- 
pocket  estimates,  there  is  a  good  chance  that  by  the  time 

you  are  ready  to  plan  a  phase  of  OT&E  in  detail,  the  budgeting 
system  has  got  you  boxed  in  —  your  flexibility  is  gone. 
Remember  this  when  you  work  on  TEMPs . ) 

909 .  Early  Termination 

a.  Occasionally,  some  aspect  of  the  system  proves  to  be 
so  poor  during  OT&E  that  completion  of  all  the  testing 
called  for  in  the  Test  Plan  would  be  wasetful.  Consider, 
for  example,  a  system  with  an  OPEVAL  MTBF  threshold  of  300 


Table  9-4 


hours  that  breaks  in  eight  different  ways  during  the  first 
16  hours  of  OT&E  operation.  In  all  likelihood  the  system 
is  not  operationally  suitable,  and  further  te.  ng  will 
prove  nothing  different.  Consider  a  detection  system  that 
detects  once  in  the  first  20  valid  opportunities  —  odds  are 
it's  not  operationally  effective  and  never  will  be  in  its 
present  configuration.  In  each  of  these  examples,  we  are 
able  to  reach  a  negative  conclusion  on  operational  effec¬ 
tiveness  or  operational  suitability  with  much  less  data  than 
would  be  required  for  a  positive  conclusion  (an  upcheck). 
Each  of  these  examples  would  probably  result  in  a  Deficiency 
Report  (see  Section  11  of  this  Guide)  and  a  COMOPTEVFOR 
decision  to  terminate  project  operations  early. 

b.  As  an  OTD,  you  should  know  in  advance  of  testing 
under  what  conditions  a  recommendation  for  early  termination 
should  be  made  to  COMOPTEVFOR.  Your  analysts  can  help  you 
derive  conditions. 

910.  Review  of  Life  Cycle  Cost  Studies  During  OT-II  (OPEVAL) 


a.  Background.  On  occasion,  CNO  has  directed  prepara¬ 
tion  of  an  LCC  (Life  Cycle  Cost)  Study  as  a  prerequisite  to 
ASU.  As  the  name  implies,  an  LCC  Study  identifies  the  life 
costs  to  develop  and  deploy  a  system,  including  procurement, 
maintenance,  repair,  training,  and  manning  costs.  When  they 
are  required,  LCC  Studies  will  be  prepared  during  full  scale 
development,  and  should  be  available  prior  to  OPEVAL. 

b.  COMOPTEVFOR  Resoonsibilities/Reguired  Actions 


(1)  OTCs  will  ascertain  if  LCC  Studies  are  required 
on  their  projects. 


that: 


(2)  When  LCC  Studies  are  required,  OTCs  will  insure 


(a)  Appropriate  TEMPs  (Parts  IV)  include  LCC 
Study  evaluation  in  OPEVAL  objectives. 

(b)  OPEVAL  Test  Plans  include  a  requirement  to 
review  the  LCC  Study  in  a  manner  similar  to  review  of  the 
ILSP  (Integrated  Logistics  Support  Plan)  as  OPEVAL  progres¬ 
ses.  This  requirement  will  be  documented  in  an  S-Test, 
"Analysis  of  Life  Cycle  Cost  Study."  The  object  of  this 
test  will  be  to  confirm  the  adequacy  of  the  assumptions  and 
analysis,  as  results  of  Tests  S-l  through  S-7  warrant. 


(Change  1) 
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(c)  A  copy  of  the  LCC  Study  is  available  prior  to 
OT-II  (OPEVAL)  project  operations. 

(3)  When  appropriate,  Evaluation  Reports  will  in¬ 
clude  assessments  of  the  adequacy  of  the  LCC  Study.  This 
assessment  will  focus  on  the  adequacy  of  the  scope  of  the 
study  and  its  assumptions  reqardinq  failure  rates,  parts 
availability,  training,  shipboard  manning,  etc.  No  attempt 
will  be  made  to  provide  a  comparative  cost  analysis,  or  to 
address  the  adequacy  of  costs. 
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Section  10 


Test  Plan  Writing 

1001.  Requirement  for  a  Test  Plan.  Test  Plans  are  required 
as  shown  in  Part  II  of  the  TEMP,  generally  before  each  iden¬ 
tified  phase  of  OT&E  (e.g.,  OT-IIIA).  This  requirement 
exists  even  though  OPTEVFOR  is  not  doing  any  hands-on  testing 
e.g.,  when  OPTEVFOR  is  monitoring  technical  testing  and 
performing  an  independent  evaluation  of  the  data.  In  this 
case,  the  OT&E  Test  Plan  would  concentrate  on: 

a.  Identifying  the  technical  test  data  to  be  provided 
COMOPTEVFOR  by  the  DA. 

b.  Describing  the  way  these  data  will  be  analyzed  to 
meet  the  objectives  of  this  phase  of  OT&E  (described  in  Part 
IV  of  the  TEMP). 


1002 .  Standardized  Suitability  Tests 


a. 

Plans. 


Seven  S-tests 
They  are: 


(1) 

Test 

S-l 

(2) 

Test 

S-2 

(3) 

Test 

S-3 

(4) 

Test 

S-4 

(5) 

Test 

S-5 

(6) 

Test 

S-6 

(7) 

Test 

S-7 

are  standardized  in  OPTEVFOR  Test 

Reliability. 

Maintainabi 1 i ty . 

Availability. 

Logistics  Supportability. 
Compatibility. 

I nter oper ab i 1 i ty . 

Training  Requirements. 


b.  As  discussed  in  Section  9,  all  of  these  standard 
tests  will  usually  be  applicable  to  OPEVALs.  Some  may  not 
be  appropriate  to  very  early  IOT&E  (e.g..  Test  S-l)  or  to 
late  FOT&E  (e.g..  Test  S-7).  In  these  cases,  do  not  use  the 
inappropriate  test(s),  but  do  not  change  the  test  numbers  of 
those  which  are  used  ( i . e . ,  Maintainability  is  always  Test 
S-2  even  if  Test  S-l  is  not  used).  Additional  tests  (S-8, 
etc. )  may  be  used  as  required. 
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c.  Note  that  tests  are  designed  to  address  projected 
characteristics  of  new  weapons  systems.  For  example.  Test 
S-4  is  supposed  to  assess  the  logistics  supportability  of 
the  weapons  system  when  it  is  deployed.  It  is  not  to  assess 
the  adequacy  of  pre-positioned  OPEVAL  spares,  or  any  other 
factors  peculiar  to  the  testing.  Similarly,  Test  S-7  addres 
ses  the  training  requirements  for  operation  and  maintenance 
of  the  weapons  system,  not  the  adequacy  of  factory  training 
for  OPEVAL  personnel. 

d.  The  standardized  list  by  no  means  exhausts  the  pos¬ 
sibilities  of  proper  suitability  tests.  These  are  only  the 
ones  that  have  permanent  numbers  assigned.  In  your  test 
plan  you  might  have  20  S-tests.  That's  OK. 

1003 .  Privacy  Act  Requirements 

a.  SECNAVINST  5211. 5A  implements  the  Privacy  Act  of 
1974  within  the  Navy.  Among  other  things,  it  defines  "per¬ 
sonal  information",  and  specifies  how  this  information  may 
be  obtained  and  maintained. 


b.  OPTEVFOR  Test  Plans  routinely  ask  operators/main¬ 
tenance  personnel  to  provide  the  following  kinds  of  infor¬ 
mation  on  forms  or  questionnaires: 

(1)  Name  of  person  completing  the  form. 

(2)  Military  experience  and/or  experience  with  the 
equipment  under  test  (e.g.,  rank/rate,  time  in  service, 
formal  schooling  on  the  equipment). 

(3)  Opinions  regarding  aspects  of  the  equipment 
(e.g.,  were  trouble-shooting  procedures  adequate?). 

c.  According  to  SECNAVINST  5211.5A,  operators/mainte¬ 
nance  personnel  are  not  providing  "personal  information" 
when  they  fill  in  their  names,  information  about  their 
experience,  and  opinions  about  the  equipment  under  test. 

This  information  may  be  requested  on  OT&E  forms  and  ques¬ 
tionnaires  without  the  necessity  for  special  procedures  or 
"Privacy  Act"  statements. 

d.  Social  Security  Numbers  are  considered  "personal 
information"  and  should  not  normally  be  requested  on  OT&E 
forms/questionnaires.  If  special  circumstances  should  make 
them  necessary,  contact  the  Administrative  Officer  (Code  11) 
for  specific  guidance  on  SECNAVINST  5211. 5A  procedures. 


1004.  Release  of  Information  to  the  Press  During  OT&E 

a.  Background .  From  time  to  time,  OTCs/OTDs  receive 
requests  from  the  press  (newspapers,  magazines,  television, 
etc.)  for  information  on  planned  or  ongoing  OT&E.  These 
requests  occasionally  include  requests  to  observe  and/or 
film  aspects  of  test  operations.  The  Commander  desires  that 
requests  from  the  press  be  processed  in  a  pre-planned  manner 
that  ensures  prompt  attention  while  avoiding  release  of  inap¬ 
propriate  material  or  interference  with  test  operations. 

b.  Organizational  Authorities  and  Responsibilities 

(1)  All  organizations/activities  involved  in  release 
of  OT&E  information  to  the  press  are  responsible  for  ensur¬ 
ing  that  security  is  maintained  —  including  special  provi¬ 
sions  of  OPSEC  (operations  security).  (See  paragraph  1007.) 

(2)  The  authority  to  specify  the  type  of  information 
relating  to  planned  or  ongoing  OT&E  that  may  be  released  to 
the  press  resides  in  Washington  --  with  the  appropriate  SYS- 
COM,  Program  Manager,  CNM,  or  OPNAV. 

(3)  After  release  is  authorized,  the  responsibility 
for  providing  actual  OT&E  details  (e.g.,  test  scenarios, 
test  results)  is  COMOPTEVFOR ' s . 

(4)  Visit  requests  to  view  operational  testing  re¬ 
quire  the  approval  of  the  operational  commander  after: 

(a)  The  appropriate  Washington  office  has  ap¬ 
proved  the  release  of  OT&E  information  that  will  be  avail¬ 
able  during  the  visit. 

(b)  COMOPTEVFOR  has  determined  that  the  visit 
will  not  affect  the  conduct  of  the  testing. 

c.  OTC/OTD  Responsibilities .  Cognizant  OTCs/OTDs  should 
assume  that  the  press  wili  request  information  on  each  phase 
of  OT&E.  They  should  then  act  on  this  assumption  during  the 
planning  stage  preceding  test  operations,  to  ensure  that 
press  requests,  when  received,  are  processed  promptly  and 
properly.  OTCs/OTDs  should: 

(1)  Determine,  from  the  DA,  what  office  has  the 
authority  to  specify  information  that  may  be  released  to  the 
press . 
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(2)  Determine,  from  the  office  of  authority,  what 
type  of  information  might  be  released  to  the  press  if  it 
were  requested.  From  this  same  office,  determine  the  like¬ 
lihood  that  press  requests  to  view  test  operations  will  be 
considered  favorably. 

(3)  Ensure,  through  informal  liaison  if  possible  but 
through  official  correspondence  if  necessary,  that  the  DA 
and  the  office  with  approval  authority  for  press  releases 
understand  and  agree  that  actual  OT&E  details  require  COM- 
OPTEVFOR  approval  prior  to  release.  (Show  them  paragraph 
205,  Disclosure  Policy  Regarding  Test  Information,  on  page 
10-18.)  Ensure  that  the  DA  understands  that  this  restric¬ 
tion  applies  to  raw  OT&E  test  data  provided  to  the  DA.  (See 
paragraph  1104. ) 

(4)  Determine  if  any  aspects  of  OT&E  test  operations 
can  be  given  automatic  COMOPTEVFOR  approval  for  release. 

These  aspects  might  include: 

(a)  Details  of  test  objectives,  test  procedures, 
scenarios,  and  methods  of  analysis  documented  in  approved 
COMOPTEVFOR  Test  Plans. 

(b)  Major  test  events  that  would  normally  be  of 
interest  to  the  press  (e.g.,  major  missile  launches,  serious 
aiecraft  accidents).  (Mote  that  this  refers  only  to  the 
event  (e.g.,  TOMAHAWK  launch),  not  to  an  evaluation  of  the 
event  (e.g.,  successful  flight). 

(5)  If  there  are  aspects  of  OT&E  test  operations  that 
can  be  given  automatic  COMOPTEVFOR  approval  for  release, 
document  them  in  memos  for  the  record. 

(6)  Include  in  the  Test  Plan  a  "Press  Release"  point- 
of-contact  —  the  designated  individual  from  the  office  of 
authority,  as  determined  above.  (See  paragraph  207  of  the 
sample  Test  Plan  on  page  10-19.) 

1005.  Format  of  OPTEVFOR  Test  Plans.  Because  of  the  enor¬ 
mous  differences  in  the  systems  undergoing  OT&E  (e.g.,  an 
LHA  on  the  one  hand,  an  angle-rate  bombing  system  on  the 
other)  and  the  ways  in  which  they  are  tested  (e.g.,  a  highly 
instrumented  evaluation  of  an  airborne  countermeasures 
system  versus  a  questionnaire-based  evaluation  of  a  new 
flight  suit).  Test  Plans  may  vary  significantly  in  layout. 

The  best  layout  is  the  one  most  usable  by  the  people  invol¬ 
ved;  the  testers,  the  data  collectors,  and  the  evaluators. 
Sample  formats  are  available  to  the  OTD  in 
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a.  Recently  published  Test  Plans  (particularly  those  on 
similar  systems  in  the  same  phase  of  OT). 

b.  The  sample  Test  Plan  used  in  the  OTD  Qualification 
Course . 


c.  The  following  pages.  In  these  pages,  samples  of 
text  are  provided  in  regular  typeface;  explanatory  remarks 
and  comments  are  provided  in  italic  type. 


10-4a 


PAGE  10-4b,  REVERSE,  BLANK 

( Change  1) 


DEPARTMENT  OF  THE  NAVY 

COMMANDER  OPERATIONAL  TEST  AND  EVALUATION  FORCE 
NORFOLK,  VIRGINIA  .23511 
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3960  ( 999-OT-IB) 
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CLASSIFICATION  (Unclassified  upon  removal  of  enclosure  (1))* 

From:  Commander  Operational  Test  and  Evaluation  Force 
To:  Distribution  List 

Subj:  Test  Plan  for  CNO  Project  999-OT-IB,  Initial  Opera¬ 
tional  Evaluation  of  the  . (See  Note  1) 

Enel:  (1)  (Classification)  COMOPTEVFOR  Test  Plan  for 

Project  999-OT-IB (*) 

1.  The  Test  Plan  for  CNO  Project  999-OT-IB  is  promulgated 
as  enclosure  ( 1) . 

2.  COMSUB _  concurrence  in  submarine  safety  aspects  of 

this  Test  Flan  is  requested  as  soon  as  possible.  (See  Note.  2) 

3.  All  addressees  are  requested  to  review  this  Test  Plan. 

Comments  are  requested  prior  to  the  planned  commencement  of 
project  operations,  1  June  1978. 

4.  Aspects  of  Project  999-OT-IB  are  classified  and  subject  to  hos 
tile  exploitation.  Consult  enclosure  (1),  Section  7,  Security, 
before  discussing  this  project  or  participating  in  pro3ect  opera¬ 
tions.  (See  Note  3) 


Distribution: 

Type  Commander 
OpeKattonal  C ommanden 
Unit  Commanden 
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T an. get  Unit 4 
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CNO  [0P-9S3) 
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(OPEVAL) ,  Operational  Evaluation  oi  the  . "  for  0 T-III 

6  IV,  me  ”CN0  Project  999-0T- III  or  IV,  Follow-on  Operational 
Evaluation  oi  the . r 

NOTE  2:  Thl 6  paragraph  16  required  ior  any  Te*t  Plan  Involv¬ 
ing  a  U.S.  submarine  In  any  capacity  ( project  thlp,  acomtlc 
target,  etc.).  Spediy  either  or  both  COMSUBLANT  and  COM- 
SUBPAC,  depending  on  the  tubmarlnelA)  Involved. 

NOTE  3:  Omit  ior  entirely  unclaimliled  project 4. 
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510  --  Test  S-9,  Human  Factors  5*9 

511  —  Test  S-10,  Safety  5-9 
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ii  CLASSIFICATION* 

10-9  (Change  4) 


PRF  Pulse  Repetition  Frequency 

All  acronyms  or  abbreviations  that  are  used  in  the  Test  Plan 
should  be  defined  here3  except 

(1)  Acronyms  for  naval  activities  included  in  the  Standard 
Naval  Distribution  List  (which  includes  almost  every  activ¬ 
ity)  need  not  be  listed. 

(2)  It  is  not  necessary  to  define  standard  metric  symbols  or 
U.S.  customary  unit  abbreviations  unless  clarity  requires 
it. 
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Section  1 

Introduction  to  the  Project  (*) 

101.  (*)  Purpose.  The  purpose  of  CNO  Project  999-OT-IB  is 

to  assess  the  potential  operational  effectiveness  and  opera¬ 
tional  suitability  of  the  . . . ,  and  its  readiness  for 

full-scale  development. 

Thi*  paKa.gKa.ph  contain*  a  bKiei  *tatement  o{  the  Kea*on 
ioK  thi *  pha*e  oi  OTS E. 

Thi*  paKagKaph  i*  the  bait*  ioK  paKagKaph  1  oi  the 
Evaluation  RepoKt. 

102.  (*)  Equipment  (or  System)  Description.  The  .  is 

a  one-way  acoustic  signaling  system  for  recall  of  UDT/SEAL 
(Underwater  Demolition  Team/Sea,  Air,  and  Land)  swimmers  in 
training  operations.  It  consists  of  an  underwater  transmit¬ 
ter  carried  in  the  recovery  boat,  and  individual  receivers 
carried  by  the  swimmers.  The  version  to  be  tested  is  an  ADM 


(advanced  development  model)  functionally  identical  to  the 
proposed  design,  but  not  representative  of  that  design  in 
size,  weight,  reliability,  or  maintainability  characteristics. 
Physical  characteristics  of  the  ADM  are  shown  in  Table  1-1. 

Thi *  paKagKaph  pKovide *  a  bKiei  *tatement  oi  the  iunc- 
tional  chaKacteKi*tic*  oi  the  end  item.  F ok  OT-J  pKoject *, 
thi*  *  ho  aid  be  iollowed  by  comment*  on  any  *igniiicant  diHex- 
ence*  between  the  te*t  item  and  the  end  item.  Fok  0T- II,  III, 
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and  IV  pxojects,  thexe  4h.ou.ld  be  no  44.gnllJLc.ant  di{{exences 
between  the  te4t  item  and  end  item.  J{  thexe  axe ,  List  them 
bxie{ly  in  Limitations  to  Scope  ( Section  3). 

The  in{oxmation  pxesented  i4  not  intended  to  4ub4titute  {ox 
ox  duplicate  in{oxmation  pxovided  opexatoX4  ox  maintenance 
pex4onnel  in  technical  documentation  iswitchology ,  etc . ) . 

(tlx ite  the  te4t4  {ixst,  then  include  only  that  detail  ne ce4 4 axy 
to  an  undexstanding  o{  the  te4t4. 

This  paxagxaph  is  the  ba4is  {ox  paxagxaph  l  o{  the  Eval¬ 
uation  R epoxt,  and  {ox  paxagxaph  101 ,  Vescxiption  o{  Equip¬ 
ment,  o{  enclo4uxe  ID  to  that  xepoxt. 

When  appxopxiate,  include  subheadings  such  as  " Mainte¬ 
nance  and  Suppoxt  Concepts"  and  " fexsonnel  and  Txaining."  List 
the  technical  manuals  to  be  evaluated. 

103.  (*)  Background 

a.  (*)  The  .  was  developed  to  satisfy  a  require¬ 

ment  of  Specific  Operational  Requirement  38-01  for  a  safe# 
reliable  recall  system  for  use  in  training  operations. 

Existing  recall  systems  use  explosive  devices.  Because 
explosive  devices  are  a  hazard  to  swimmers  in  the  water, 

development  of  the  .  concentrated  on  electronically 

generated  acoustic  signals. 

b.  (*)  IOT&E  (initial  operational  test  and  evaluation) 


of  the  .  began  under  Project  999-OT-IA,  conducted  from 

. to .  The  purpose  of  OT-IA  was  to  assess  the 
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potential  operational  effectiveness  and  operational  suits  - 

bility  of  three  competing  designs  of  the . ,  in  order  to 

assist  the  Developing  Agency  in  selecting  between  them.  As 
a  result  of  OT-IA,  COMOPTEVFOR  found  the  system  manufac¬ 
tured  by  .  to  have  the  most  potential,  and  recommended 

certain  changes  to  his  functional  design.  These,  and  other 
changes,  have  been  incorporated  into  the  ADM  to  be  tested  in 
OT-lB. 

This  paAagAaph  concisely  summaAize*  the  majoA  event* 
{emphasizing  previous  0T6 EJ  that  led  to  this  testing.  The 
TEMP  is  the  souacc  oi  the  inioAmation  summaAized  heaein. 

This  paAagAaph  is  the  basis  ioA  paAagAaph  3  o&  the  Eval¬ 
uation  R epoAt  and  ioA  Section  t  oi  enclosuAe  (I)  to  that 
a epoAt. 
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Section  2 

Administrative  Information  (*) 

201.  (*)  General.  General  responsibilities  of  activities 
involved  in  this  testing  are  provided  in  this  section,  as 
well  as  appropriate  points  of  contact.  Continuing  close 
liaison  is  essential  to  timely  and  successful  prosecution  of 
this  project. 

202.  (*)  Responsibilities 
a.  (*)  COMOPTEVFOR 

(1)  (*)  Promulgate  major  changes  to  this  Test  Plan. 

(2)  (*)  Coordinate  arrangements  for  fleet  services. 

(3)  (*)  Conduct  briefings  for  all  participating 
units,  including  OPSEC  (operations  security)  requirements 
and  procedures. 

(4)  (*)  Issue  appropriate  OPORDs  (Operation  Orders) . 

(5)  (*)  Provide  failure  and  failed  part  data  to  the 
Developing  Agency  as  soon  as  possible. 

(6)  (*)  Analyze  test  results  and  publish  appropriate 

reports . 

(7)  (*)  Others  ae  necessary . 

If  the  project  is  reassigned  for  prosecutiont  provide 
separate  subparagraphs  outlining  the  responsibilities  of  the 
Headquarters  and  of  the  proseouting  agency. 


b.  (*)  Developing  Agency 
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(1)  (*)  Furnish  required  material  and  technical 

support. 

(2)  (*)  Provide  required  plans  and  funding  for 
installation/removal  of  project  equipment. 

(3)  (*)  Provide  TYCOM  and  Unit  commanders  with  data 
on  the  impact  of  the  test  installation  on  operational  capa¬ 
bilities  of  the  unit  providing  services. 

(4)  (*)  Provide  for  required  training  of  fleet  and 
OPTEVFOR  personnel  in  operation  and  maintenance  of  the 
equipment . 

(5)  (*)  Provide  copies  of  Failure  Analysis  Reports 
to  OPTEVFOR. 

(6)  (*)  Provide  funding  for  (identify  any  other 
support  required ,  i.e .,  data  reduction,  reconstruction, 
simulation,  etc.). 

(7)  (*)  Provide  for  appropriate  safety  certifications. 

(8)  (*)  Certify  equipment  ready  for  OPEVAL  in  accord¬ 
ance  with  OPNAVINST  3960.10.  ( OPEVAL  only) 

(9)  (*)  Others  as  necessary . 

c.  (*)  Type  Commander  is  requested  to  direct  the  assigned 
project  submarine  to: 

(1)  (*)  Make  personnel  available  for  iquired  training. 

(2)  (*)  Operate  in  accordance  with  this  Test  Plan 
and  COMOPTEVFOR  OPORDs. 
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(3)  (*)  Maintain  installed  equipment  and  ensure 
availability  of  trained  personnel  to  operate  and  maintain 
the  equipment. 

(4)  (*)  Support  the  data  recording  requirements  of 
this  Test  Plan. 

(5)  (*)  Keep  COMOPTEVFOR  informed  of  any  condition 
which  may  affect  prosecution  of  this  project. 

(6)  (*)  Prepare  and  submit  reports  in  accordance 
with  Section  6. 

(7)  (*)  Othuu  <u  neceAiaJt,y . 

203.  (*)  Points  of  Contact  [ThiA  information  iA  often 


COMOPTEVFOR 

LCDR  Charles  BROWN 
Operational  Test  Coordinator 
Staff,  COMOPTEVFOR  (Code  46) 
Norfolk,  VA.  23511 
Autovon  690-4051 
Telephone  804-444-4051 

LT  James  A.  KING 
Operational  Test  Director 
Staff,  COMOPTEVFOR  (Code  462) 
Norfolk,  VA.  23511 
Autovon  690-4051 
Telephone  804-444-4051 

NAVSYSCOM  {Developing  Agency] 

CDR  T.B.  SUTHERLAND 
Acquisition  Manager 
NAVSYSCOM  (PM- 30 3) 

Washington,  D.C.  20360 
Autovon  222-8590 
Telephone  202-692-8590 


tabulated] 
a.  (*) 
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c.  (*)  Othe.su  cu  Ke.quln.ed 

204.  (*)  Visitor  Control.  Approvals  for  visitors/ship 
riders  associated  with  this  testing  will  be  kept  to  a  minimum 
because  of  limited  space  available.  Visit/rider  authorization 
will  be  granted  for  valid  requirements/  for  technical  assistance 
requested  by  the  OTD  (Operational  Test  Director) ,  or  on  a  gen¬ 
uine  need-to-know  basis.  Requests  for  visits/riders  during 
project  operations  will  be  addressed  to  COMOPTEVFOR,  info  [unit 
commanding  oHIcck).  COMOPTEVFOR  will  coordinate  the  requests 
with  [unit  admlnhtKatlve  commcndeK  and  unit  commanding  o^lceK) . 
Affirmative  response  by  COMOPTEVFOR  must  be  received  before 
visits  are  authorized. 

205.  (*)  Disclosure  Policy 

a.  (*)  Test  Information.  No  test  data,  message,  correspon¬ 
dence,  briefing,  or  statement  stating  conjecture,  opinion,  conclu¬ 
sions,  or  recommendations  regarding  this  testing  will  be  directed 
outside  OPTEVFOR  without  prior  COMOPTEVFOR  approval.  Messages 
involving  immediate  safety  are  excluded  from  this  restriction. 

b.  Proprietary  Information.  Proprietary  information  will 
not  be  disclosed  by  COMOPTEVFOR.  Requests  for  access  to  such 
information  will  be  referred  to  the  proprietor  agency  for  dis¬ 
position. 

206.  (*)  Deviations  from  the  Test  Plan.  The  OTD  is  author¬ 
ized  to  deviate  from  this  Test  Plan  as  the  operational 
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situation  and  good  judgement  dictate,  keeping  COMOPTEVFOR  advised. 

207 •  Release  of  Information  to  the  Press 

a.  Any  requests  from  the  press  (newspapers,  television,  etc.) 
for  information  about  this  testing  (including  requests  to  observe 
or  film  aspects  of  testing)  will  be  referred  to  UmeAt  the  "?> ie*6 
Release  point- o^-co ntact ' 6  code,  a&  diicuned  in  pa.Aa.gAa.ph  1004 
0((  thi*  Guide) ,  who  will  specify  what  information  may  be  released 
or  what  aspects  of  testing  may  be  observed  or  filmed. 

b.  Information  to  be  provided  to  the  press  will  be  prepared 
under  the  direction  of  the  OTC  (Operational  Test  Coordinator) ,  who 
will  ensure  that  it: 

(1)  Is  within  the  framework  of  approval  specified  by  [inteAt 
the  "?Ae*&  Release”  point- o i- con.ta.ct' &  code). 

(2)  conforms  to  the  security  guidelines  of  Section  7  of  this 
Test  Plan  (including  OPSEC  requirements) . 

(3)  Is  approved  for  release  by  COMOPTEVFOR. 

c.  Requests  to  observe  or  film  aspects  of  testing  that  are 
approved  by  [inAeAt  the  "PAeti  Retea&e"  point- o$- contact' 4  code) 
will  be  considered  visit  requests  on  a  genuine  need-to-know  basis, 
for  the  purpose  of  visitor  control  (paragraph  204  above) .  COMOPTEVFOR 
will  approve  such  visit  requests  if  they  do  not: 

(1)  Interfere  in  any  way  with  test  operations. 

(2)  Jeopardize  security  (including  OPSEC). 
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Section  3 

Scope  of  the  Evaluation  (*) 

301.  (*)  Objectives.  The  objectives  of  Project  999-OT-IB  are  to 

determine : 


a . 

b . 

The le  ant  the  objective*  o£  thl*  phcue  o£  OTSE.  They  axe  the 
appAopAlate  "OTSE  Objective *"  o£  Poa t  IV  o£  the  TEMP. 

Objective *  that  addAe **  opeAatlonat  e££ectlvene **  aAe  lilted 
ilAit,  iottovaed  by  objective*  that  addAe**  opeAatlonat  Aultablllty . 
"OpeAatlonat  e££ectlvene**"  and  "opeAatlonat  Aultabltlty"  may  be 
u*ed  a*  *ub heading* ,  bat  Ahoutd  not  appeaA  In  the  actual  woAdlng 
o£  objective*. 

Thl*  paAagAaph  1*  the  ba*l*  £oA  paAagAaph  4a  o£  the  Evaluation 
Repo At. 

302.  (*)  Evaluation  Criteria.  CNO  provided  the  following  criteria 
in  reference  (a) : 

Ll*t  the  thAe*hotd*  e*tabtl*hed  £oa  thl*  pha*e  o£  OTSE  In  PaAt 
I  o£  the  TEMP  WequlAed  OpeAatlonat  ChaAacteAhtlc*  \ ,  oa  £Aom  the 
OR,  NVCP,  oa  Aetated  pAogAam  document.  In  any  c a*e,  Identify  the 
AouAce  o£  the  c AlteAla.  Vo  not  Include  cAiteAla  that  only  Aepeat 
objective* . 

Thl*  paAagAaph  1*  the  ba*l*  £oA  paAagAaph  4b  o£  the  Evaluation 
RepoAt,  and  £oa  paAagAaph  301  o£  enctoAUAe  ( I )  to  that  AepoAt. 

303.  (*)  Testing.  Test  operations  will  exercise  the . in 

realistic  scenarios,  in  representive  operational  environments . 

These  operations  will  provide  the  data  for  evaluation  in  individual 
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.j'&A  tests  of  operational  effectiveness  (E-tests)  and  operational  suite- 
bility  ( S- tests)  discussed  in  Sections  4  and  5. 

Thin  nectlon  nummanlzen  the  te.6t4.ng  that  villi  generate  data  £oK 
evaluation ,  and  general  pnoeedunen  to  be  uned.  When  the  VA  In  In 
change  oh  the  testing.  Including  hactony  tenth  ok  demonhtnatlonh , 
nehcnence  to  hlh  tent  plan  with  a  bnleh  dencnlptlon  oh  the  tenting 
In  appKopnlate.  Vehlne  the  data  to  be  collected  by  the  VA  and 
hunnlnhed  to  OPTEVFOR.  When  COHOPTEVFOR  In  In  change  oh  tenting , 
the  h°Howlng  panagnaphn  pnovlde  genenal  guidance  to  tent  pantlcl- 
pantn . 

a.  (*)  Safety.  In  the  conduct  of  all  operations  associated  with 
this  project,  SAFETY  IS  PARAMOUNT.  No  operations  will  be  conducted 
that,  in  the  opinion  of  the  Commanding  Officer  concerned  or  the 

m  OTD,  will  endanger  personnel  or  equipment.  In  an  unsafe  situation 
should  develop,  appropriate  corrective  action  will  be  taken  immedi¬ 
ately.  COMOPTEVFOR  will  be  notified  as  soon  as  possible  of  the 
circumstances,  including  rectifying  procedures  initiated  and  recom¬ 
mended  further  action. 

b.  (*)  Range  Procedures 

Thin  panagnaph  dlncunnen  npeclal  pnoeedunen ,  Inntnumentatlon , 
communlcatlonn ,  etc.,  that  may  be  nequlned  when  openatlonh  ane 
conducted  ( In  whole  on  In  pant)  on  a  nange.  Make  nehenence  to 
appnopnlate  nange  manualn  on  Inntnuctlonn ,  an  welt  an  to  any  bnleh- 
Ingn  nequlned  behone  nange  openatlonn . 

c.  (*)  OPORDs  and  Exercise  Messages 

The  OTP  may  be  nequlned  to  pnepane  an  OPORV  behone  pnoject 
openatlonn .  Any  npeclal  Inntnuctlonn  that  will  be  contained  In  the 
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OPORV  should  be  dh  culled  In  thh  paA.agA.aph.  kilo,  the  OTV  may  be 
Aequ.lA.ed  to  pAepaAe  Exexche  Menage*  ^ok  each  day' i  opexatloni  at 
lea.  Paxtlclpatlng  unlti  ihould  be  advhed  hexe  of  any  ipeclal 
Initxuctloni  theie  menagei  will  pxovlde  l  how  a  uni  will  be  Identified, 
OPS  EC  InitAuctloni,  etc. ) . 
d.  (*)  Data  Collection 

(1)  (*)  Data  Sheets.  Special  data  sheets  for  use  in 
this  testing  are  contained  in  Annex  3 .  Copies  will  be  distri¬ 
buted  to  test  participants  by  the  OTD.  Standard  Navy  forms, 
logs,  etc.  that  will  supplement  these  data  sheets  are 
identified  in  Sections  4  and  5. 

(2)  {*)  Automatic  Data  Recording.  The  following 
automatic  data  recording  systems  will  be  employed  throughout 
test  operations: 

304.  (*)  Limitations  to  Scope 

Lilt  the  ilgnlflcant  facto xi  that  will  [ox  pxobably 
will)  pxevent  complete  accomplhhment  of  the  puxpoie  of  thh 
phaie  of  teitlng.  Typical  factoxi  axe  taxget  chaxactexh- 
tlci  not  fully  xepxeientatlve  of  the  thxeat,  teit  axea 
chaxactexhtlci  not  xepxeientatlve  of  the  expected  opexa- 
tlonal  envlxonment,  ox  depaxtuxei  fxom  opexatlonal  xealhm 
cauied  by  teit  condltlom.  Include  In  the  llmltatlom 
itatement  any  woxk-axound  pxoceduxei  being  planned  to  xeduce 
the  effecti  of  the  limiting  factoxi.  Pox  example,  "Avail¬ 
able  taxgeti  do  not  xepxeient  xealhtlc  thxeati.  Howevex, 
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I  puter  1imula.tJ.01u  to  tran*late  tout  re*ult*  into  operational 

i 

j  MOE* .  • 

i  Thi*  paragraph ,  modified,  a*  a  re*  alt  ol  actual  condi¬ 

tion*  that  exiuted  during  testing,  provide*  the  ba*i*  lor 
paragraph  4c  ol  the  Evaluation  Report  and  lor  paragraph  S 04 
ol  enclo*ure  (J)  to  that  report . 


i 

t 

i 

! 

) 

I 


I 
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Section  4 

Operational  Effectiveness  (*) 

401.  (*)  Scenarios.  Effectiveness  testing  will  exercise 
.  in  realistic  scenarios  in  typical  operational  envi¬ 
ronments.  These  scenarios  are  described  below.  Plans  and 
geometries  for  specific  runs  to  simulate  these  scenarios  are 
described  in  Annex  A.  (Alternatively:  Plans  and  geometries 
for  specific  runs  to  simulate  these  scenarios  are  described 
below  in  the  procedures  for  individual  tests.) 

a.  (*)  Scenario  A,  Barrier  Patrol  . 

b.  (*)  Scenario  By  Amphibious  Assault  . 

This  paragraph  describee  the  operational  ecenarioe  in 

which  the  equipment  will  be  exercised  to  determine  its  mis¬ 
sion  effectiveness  or  to  define  tactics.  One  scenario  may 
suffice  for  single-mission  equipment ;  several  will  be 
required  for  multi-mission  equipment. 

In  each  scenario  description ,  state  the  operational  mis¬ 
sion  being  simulated ,  and  describe  the  actions  of  simulated 
friendly  and  threat  participants t  but  not  the  actions  of 
units  merely  monitoring  or  providing  instrumentation. 

Support  unit  instructions  are  provided  in  run  plans. 

402.  (*)  Test  E-l,  Recall  Envelope 

a.  (*)  Object.  To  determine  the  ranges  and  depths  at 
which  swimmers  can  . 
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The  object  of  an  E-test  is  to  aeeeee  some  aspect  of  a 
project  objective  ort  on  oocasiont  more  than  one  project 
objective.  For  simple  projects t  the  test  object  and  project 
objective  may  be  identical. 

b.  (*)  Procedure.  Recall  signals  will  be  transmitted 

to  swimmers  at  range,  depth,  and  sea  state  combinations 
shown  in  . 

Identify  the  scenarios  and  runs  which  provide  data  for 
this  testj  the  test  variables  involved ,  and  the  necessary 
sample  size.  The  information  in  this  paragraph  should 
complement  (not  repeat)  information  contained  in  run  plans. 

c.  (*)  Data  Requirements.  Identify  the  data  required 
for  this  test. 

d.  (*)  Data  Analysis.  Data  Sheet  E-l  will  be  used  to 
construct  curves  defining  the  boundary  of  90%  probability  of 


Describe  how  the  data  will  be  analyzed  and  how  the 
results  are  intended  to  be  presented  (e.g.t  ohart3  plott  or 
specific  number).  Identify  analytic  methods  peculiar  to 
this  test.  When  appropriate ,  carefully  define  such  cate¬ 
gories  as  "No-Test"  or  "Failure . " 

At  times t  a.  separate  annex  describing  general  analytical 
methods  or  presenting  analytical  details  may  be  appropriate 
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(see  Annex  C).  In  those  oases,  reference  the  annex  in  the 
appropriate  test  or  tests,  or  introduce  it  in  paragraph  401. 

403.  (*)  Test  E-2/ .  If  desired,  begin  on  a  new 

right-hand  page. 

When  Oomputer  simulations  are  used  to  extend  the  data 
base,  it  is  necessary  to  describe  (under  Procedure)  the  com¬ 
puter  model  and  the  means  by  which  it  was  (or  will  be) 
proven  to  reproduce  the  operational  situation  adequately. 

404.  (*)  Test  E-3,  Survivability/Vulnerability 

a.  (*)  Object.  To  assess  the  characteristics  of  the 

....  and  its  installation  which  might  lead  to  major  or  total 
degradation  in  mission  performance  because  of  enemy  weaponry. 
The  objective  is  to  avoid  a  "cheap  kill"  whereby  a  severed 
cable,  a  shock-damaged  switchboard,  or  computer  casualty  can 
eliminate  the  .  as  an  effective  combat  system. 

b.  (*)  Procedure .  On-scene  observations  will  be  made 

by  the  OTD  using  vulnerability  checklists  and  functional 
block  diagrams  as  guides.  The  OTD  will  physically  trace  the 
. .  from  inputs  to  outputs,  considering  the  following: 

(1)  (*)  Primary  and  secondary  effects  of  weaponry 
will  be  considered,  including  conventional,  nuclear  (includ¬ 
ing  EMP) ,  biological,  chemical,  and  laser  weapons.  Effects 
include  material  damage  and  crew  casualties. 
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(2)  (*)  Subsystems  may  be  considered  vulnerable 


because  of: 


(a)  (*)  Unnecessary  number  of  components,  large 


size,  or  large  (vulnerable)  area. 


(b)  (*)  Basic  structure  not  hardened,  shielded. 


or  armored  to  resist  penetration. 


(c)  (*)  Insufficient  redundancy  of  critical  com¬ 
ponents  or  cable  pathways. 

(d)  (*)  Electronics  mounted  external  to  the  skin 
of  the  ship  susceptible  to  blast,  shock,  or  fragmentation. 

(e)  (*)  Electronics  using  solid-state  electro¬ 
nics  without  coupling  protection,  etc.,  against  EMP. 


(f)  (*)  Lack  of  manual  inputs  and/or  manual 


override. 


(3)  (*)  The  on-board  installation  will  be  examined 
to  determine  that  the  following  good  survivability  practices 
have  been  followed: 

(a)  (*)  Critical  components  and  series  compo¬ 
nents  are  installed  close  together. 

(b)  (*)  Critical  areas  are  shielded  by  non- 
critical  components  and/or  armor. 

(c)  (*)  Parallel  or  redundant  components  are 
diffused  or  installed  far  apart  (at  least  two  damage  radii). 
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(d)  (*)  Hazardous  material  is  isolated  from  vul¬ 
nerable  subsystems  and  systems. 

(4)  (*)  Results  of  specific  target  analyses,  surviva¬ 
bility  models,  and  degraded  mode  effectiveness  testing,  will 
be  examined  to ‘determine  the  degree  of  vulnerability  of  the 
various  subsystems,  taking  the  planned  installation  into 
account. 

c.  (*)  Data  Requirements.  The  OTD  will  record  results 
of  his  observations  in  the  OTD  Journal  and  on  checklists. 

d.  (*)  Data  Analysis.  On-scene  observations  and  check¬ 
list  data  will  be  assessed  qualitatively,  taking  the  likely 
threats  into  account.  Personnel  safety,  damage  control,  and 
casualty  mode  effectiveness  will  be  included. 

(1)  (*)  Critical  subsystems  considered  unnecessarily 
vulnerable  will  be  pinpointed  for  the  evaluation  report. 
Changes  to  reduce  the  "cheap  kill"  potential  will  be  sug¬ 
gested  when  possible. 

(2)  (*)  A  list  will  be  prepared  of  externally  moun¬ 
ted  subsystems  considered  vulnerable. 
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Section  5 

Operational  Suitability  (*) 

501.  (*)  General .  The  suitability  testing  will,  in  most 
instances,  use  data  generated  by  continuous  operation  of  the 
equipment  throughout  test  operations,  including  the  E-test 
runs  described  in  Section  4  and  Annex  A.  Runs  specifically 
designed  to  generate  suitability  data  are  described  below 
under  the  test  to  which  they  apply. 

502.  (*)  Test  S-l,  Reliability 

a.  (*)  Object.  To  determine  the  probability  of  com¬ 
pleting  a  miss ion /engagement  of  (specified  time  or  cycles) 
without  critical  or  major  failure. 

Define  critical,  major,  and  minor  failures  as  specifically 
as  possible .  See  this  Guide's  Glossary  for  general  defini¬ 
tions. 

b.  (*)  Procedure .  This  test  will  be  conducted  continu¬ 
ously  during  test  operations. 

c.  (*)  Data  Requirements.  Maintenance  Actions  Forms 
will  be  completed  for; 

(1)  (*)  Each  failure  or  discrepancy  noted  during 
operations . 

(2)  (*)  Each  preventive  maintenance  action  which 
finds  a  failed  part. 
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a.  (*)  Data  Analysis.  R  (mission  reliability)  is  com¬ 
puted  using  the  formula: 

R  *  eXR  <-  ®BP> 
where  t  =  mission  time 

HTBF  =  mean  time  between  failures 
For  equipment  such  as  guns  or  torpedo  tubes s  cycles  vice 
time  is  significant ,  and  the  formula  is 


R  =  exp  ( - 


MCBF 


where  C  -  nominal  cycles  per  mission/engagement 
MCBF  =  mean  cycles  between  failure 
For  one-shot  devices  such  as  missiles  or  torpedoes s 

neither  time  nor  cycles  is  appropriate ,  and  the  formula  is 

_  Valid  Successes 
Valid  Attempts 

In  this  case3  specific  definitions  on  validity  are  required. 
Care  must  be  taken  not  to  confuse  success  for  material 
purposes  with  success  for  effectiveness . 

Where  appropriate 3  include  the  following  COMOPTEVFOR 
approach  to  damage  caused  by  handling: 

(1)  Definition.  Handling  damage  is  caused  by  human 
error  during  physical  movement ,  transportation  t  or  handling 
by  authorised  personnel. 

(2)  Categories 
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(a)  Gross  handling  damage  involving  negligence ; 
all  actione  other  than  normal  action  by  competent,  knowledge¬ 
able  personnel  following  proper  written/verbal  instructions . 

(The  individual's  command  is  responsible  for  making  the 
judgement  regarding  negligence,  by  considering  the  normality 
of  the  action,  and  the  correctness  of  the  instructions 
followed. ) 

(b)  Normal  handling  damage  does  not  involve 

negligence . 

(3)  Treatment  of  Handling  Damage  in  MTBF  Calcula¬ 
tions.  Failures  resulting  from  handling  damage  are  included 
in  MTBF  calculations  unless  the  damage  is  categorized  as 
gross  handling  damage. 

503.  (*)  Test  S-2,  Maintainability 

a.  (*)  Object.  To  determine  the  maintainability  of  the 
.  in  the  intended  operational  environment. 

b.  (*)  Procedure 

c.  (*)  Data  Requirements.  Maintenance  Actions  Forms . 

To  preclude  not  being  able  to  assess  maintainability 

(e.g.,  MTFL,  MTTR,  etc)  because  no  (or  few)  failures  actu¬ 
ally  occur  during  test  operations,  make  provision  for  a 
maintenance  demonstration,  after  test  operations ,  using 
pre faulted  modules.  Include  the  procedure  here. 

d.  (*)  Data  Analysis 
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504.  (*)  Test  S-3,  Availability 

a.  (*)  Object.  To  determine  the  probability  that  the  equipment 
will  be  operationally  ready,  when  needed,  at  any  point  in  time. 

b.  {*)  Procedure 

c.  (*)  Data  Requirements.  All  operator  logs,  maintenance  action 

forms,  and  time  meters  will  be  reviewed . 

Any  special  in4tA.uct4.on4  on  the  handling  o£  data  { oam*  on.  . 
xecoKd4  4hould  be  included  a4  well  a4  any  4pecial  deiinition4 
tetimA  applicable  to  thi4  te4t. 

d.  (*)  Data  Analysis.  Operational  availability  is  computed 

using  the  formula: 

.  _  Uptime _ 

o  Optime  +  Downtime 

Any  4pecial  con4iden.ation4  that  may  make  4ome  data  invalid  on. 
be  given  le44  weight  4hould  be  included. 

505.  (*)  Test  S-4,  Logistic  Supportability  (r 

a.  (*)  Object.  To  assess  the  logistic  supportability  of  the 

.  in  a  deployed  operational  environment.  (See  Annex  6  o£  thi4 

Guide. ) 

b.  (*)  Procedure .  This  test  will  be  conducted  before  and  con¬ 
tinuously  during  project  operations. 

(1)  (*)  The  adequacy  of  the  ILSP  (Integrated  Logistic  Sup¬ 
port  Plan)  will  be  assessed.  Special  attention  will  be  given  to 
the  planning  for  delivery  of  resources  that  are  required  to  support 
the... but  are  not  available  during  OPEVAL.  [All  AuppoKt  Ke4ouKce4 
4hould  be  available  duxing  OPEVAL.  Only  in  extreme  ca4e4  Ahould 
the  OTV  accept  partial  4uppoKt. ) 

(2)  (*)  The  following  items  related  to  logistic  support  will 
be  evaluated: 
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(a)  (*)  Clarity#  completeness#  accuracy#  and  availability 
of  technical  manuals  and  PMS  (Planned  Maintenance  System)  documen¬ 
tation  . 

(b)  (*)  Availability  and  adequacy  of  test  equipment  and 
special  tools. 

(c)  (*)  Adequacy  of  the  support  (including  spare  parts# 
operating/maintenance  procedures#  and  training)  provided  in  conjunc¬ 
tion  with  test  equipment  and  special  tools. 

(d)  (*)  Effect  of  maintenance  requirements  on  manning. 

(e)  (*)  Adequacy  of  supply  support. 

1.  (*)  The  requirements  for#  and  availability  of# 
spare  parts  during  OPEVAL  will  be  evaluated.  Any  requirements  that 
indicate  unexpectedly  high  component  failure  rates  will  be  investi¬ 
gated  . 

2.  (*)  The  schedule  for  submission  of  PTD  (pro¬ 
visioning  technical  documentation)  to  the  inventory  control  point 
lelthex  the  A  v  tat  ton  Supply  OUl  ce  ox  Ship*  Paxt s  ContKol  Centex) 
will  be  evaluated. 

{This  step  Is  vital  with  xespect  to  the.  capability  oi  the.  Navy 
Supply  System  to  suppoxt  the.  system  aitex  ileet  IntKoductlon .  P TV 
Include s  the  technical  dKawlng s,  manuiactuxex' *  paxts  list s,  esti¬ 
mated  paxt s  ialluxe  Kates,  duty  cycles,  etc.,  upon  which  the  Inven- 
toxy  contKol  point  will  base  Its  s paxes  allowance  computations  and 
spaxe  paxts  purchase*.  Thexejoxe,  100 !  spaxe  paxts  availability 
duxlng  OPEVAL  Is  meaningless  ll  the  PA  has  not  pxovlded  iox  timely 
dellvexy  oi  PTV,  because  ileet  units  will  xecelve  the  system  without 
the  paxts/ equipage  lists  ox  the  spaxe  paxts  to  suppoxt  It.  Note: 
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It  u*ually  i*  IS  month*  a^tex  PTV  delivexy  be^oxe  oxganic  l Navy] 
*upply  *uppoxt  i*  available..  ) 

(£)  (*)  The  adequacy  of  the  following  aspects  of  sup¬ 
port  will  also  be  evaluated:  {Depending  on  the.  *y*tem  undo*  evalua¬ 
tion.  ) 


1.  (*)  Calibration  requirements. 

2.  (*)  Provisions  for  packaging,  handling,  storage, 
and  transportation. 


2.  (*)  Stowage  space  for  spare  parts. 

(3)  (*)  All  support  resources  used  during  testing  that  are 
not  to  be  provided  to  operational  units  will  be  noted. 

In  all  Te*t  Plan *  &ox  *y*tem* / equipment  that  xequixe  {ox  may 
xequixe)  lubxicant *,  include  undex  Te*t  S-4  a  check  to  detexmine 
ii  the  xequixed  lubxicant *  axe  *tandaxd  lubxicant*.  CNO  ha*  noted 
that  incxea*ed  xequixement*  iox  non-*tandaxd  li.e.,  *  pedal  ox 
pxopxietaxy)  lubxicant*  have  cau*ed  *  towage  pxoblem *  aboaxd  *hip 
and  have  buxdened  the  *upply  *y*tem. 
c.  (*)  Data  Requirements 

(1)  (*)  The  data  required  to  conduct  this  test  are  as  follows: 


(a)  (*)  The  ILSP. 

(b)  (*)  All  technical  manuals  and  PMS  documentation, 
in  preliminary  or  final  form. 

(c)  (*)  Preliminary  APLs/AELs  (allowance  parts/equipage 

lists) . 

(d)  (*)  All  related  test  equipment  and  special  tools. 

(e)  (*)  Completed  NAVSUP  Form  1250,  with  part  number  and 
APL  number  (or  nomenclature  of  parent  equipment) ,  for  each  spare 
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tools. 


lists. 


(2)  (*)  In  addition,  the  following  will  be  provided  by  the 


(a)  (*)  List  of  all  required  technical  manuals. 

(b)  (*)  List  of  all  required  test  equipment  and  special 


(c)  (*)  List  of  all  related  preliminary  or  interim  parts 


(d)  (*)  List  of  all  MRCs  (Maintenance  Requirement  Cards) . 

(e)  (*)  Certification  of  the  submission  of  PTD  to  the 
inventory  control  point  and  projected  dates  for  all  future  submis¬ 
sions  of  PTD. 

( The  bex* Leb  oh  libtb  will  allow  the.  OTV  to  determine  the  complete- 
neb  b  oh  the  on-boaxd  buppoxt  package,  and  provide*  him  a  checklist.) 

d.  (*)  Data  Analysis.  Logistic  support  data  will  be  analyzed 
quantitatively  and  qualitatively. 

506.  (*)  Test  S-5 ,  Compatibility 

a.  (*)  Object.  To  assess  the  compatibility  of  the .  with 

its  operating  environment. 

b.  (*)  Procedure 

c.  (*)  Data  Requirements 

d.  (*)  Data  Analysis 

I h  debined,  thib  tebt  may  be  bubdivided  ab  hollowb: 
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Tat  S- 5 A,  Physical  Compatibility 
Tat  S-  58,  functional  Compatibility 
Tat  S-SC,  ElectKonlc/ElectKlcal  Compatibility 
HoKmal  operation*  may  not  expose  InteK^eKence  ok  Incom¬ 
patibility, ,  and  the  OTV  may  have,  to  conduct  special  tests, 
operating  vaKlous  equipments  In  vokIous  modes,  to  detect  any 
potential  InteKleKence. 

507.  (*)  Test  S-6 ,  Interoperability 

a.  (*)  Object.  To  determine  the  adequacy  o£  the  inter¬ 
faces  between  the . and . 

b.  (*)  Procedure 

c.  (*)  Data  Requirements 

d.  (*)  Data  Analysis 

508.  (*)  Test  S-7f  Training  Requirements 

a.  (*)  Object.  To  assess  the  adequacy  of  the  training 
planned  for  operators  and  maintenance  personnel. 

b.  (*)  Procedure  {.See  Annex  E  to  this  Guide . ) 

c.  (*)  Data  Requirements 

d.  (*)  Data  Analysis 

509.  (*)  Test  S-8f  Documentation 


3 

^  ", 

a. 

(*)  Object.  To  assess 

the  adequacy  and 

accuracy 

of  the 

documentation  provided 

for  the  .... 

% 

b. 

(*)  Procedure 

rc* 

fc-: 

c. 

(*)  Data  Requirements 

d. 

(*)  Data  Analysis 

£ 
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510.  (*)  Test  S-9f  Human  Factors 

a.  (*)  Object.  To  assess  the  adequacy  of  human 
factors  features  of  the  .... 

b.  (*)  Procedure 

c.  (*)  Data  Requirements 

d.  (*)  Data  Analysis 

511.  (*)  Test  S-10,  Safety 

a.  (*)  Object.  To  assess  the  adequacy  of  safety 
features  of  the  .... 

b.  (*)  Procedure 

c.  (*)  Data  Requirements 

d.  (*)  Data  Analysis 
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Section  6 
Reports  (*) 

601.  (*)  General.  Reports  required  in  connection  with  this 
project  are  described  in  the  following  paragraphs.  Distri¬ 
bution  should  be  limited  where  indicated. 

602.  (*)  CASREP/CASCOR 

Unite  designated  to  conduct  or  support  testing  under 
this  project  should  he  directed  to  include  COMOPTEVFOR  as  an 
addressee  on  any  CASREP/CASCOR  messages  that  may  indicate 
any  reduction  in  the  ability  to  complete  the  mission 
required  by  this  Test  Plan. 

603.  (*)  Readiness  Reports 

a.  (*)  DA  Certification 

For  OPEVALSt  the  DA  shall  certify  readiness  in  accord¬ 
ance  with  OPNAVINST  3960.10.  For  other  OT  operationst  the 
OTD  will  ensure  that  prerequisite  technical  achievements 
have  been  satisfied  before  commencing  operations . 

b.  (*)  Unit  Readiness 

The  Commanding  Officers  of  participating  unite  shall 
report  by  message  to  COMOPTEVFOR ,  copy  to  the  operational 
commander ,  that  the  unite  are  ready  to  commence  operations . 
Any  exceptions  or  reservations  on  the  part  of  a  Commanding 
Officer  should  be  included  in  this  report.  For  OPEVALs, 
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ship  Commanding  Officers  will  report  in  accordance  with 
OPNAVIHST  3960.  10. 

604.  (*)  OPEVAL  Commencement  Report 
See  Section  11  of  thie  Guide. 

605.  (*)  Status  Reports 

a.  (*)  SITREP 

See  Section  11  of  this  Guide. 

b.  (*)  Deficiency  Reports 
See  Section  11  of  this  Guide. 

606.  (*)  Evaluation  Reports 

a.  (*)  Unit  Commander's  Report 

Unit  commanders  of  participating  and  supporting  units 
should  be  tasked  to  submit  letter  reports  to  COMOPTEVFOR , 
copy  to  their  operational  oommander,  commenting  on  their 
impressions  of  the  operational  effectiveness  and  operational 
suitability  of  the  equipment ,  tactics,  and  areas  requiring 
further  investigation. 

b.  (*)  COMOPTEVFOR  Report 

If  tests  are  prosecuted  by  a  non-Headquartere  activity , 
specify  the  time  allowed  for  thie  activity  to  submit  a  draft 
report  to  the  Headquarters.  Specify  required  Quick-Look 
and/or  Partial  Reports. 

607.  (*)  OPTEVFOR  Tactics  Guide 
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Specify  the  requirement  for  an  OPTEVFOR  Tactics  Guide  in 
the  same  manner  as  for  an  Evaluation  Report. 
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^ 

*  V-V 


V‘ 

701. 

(*)  A 

■A 

i 

a. 

(*) 

(1) 

£ 

(2) 

b. 

(*) 

1 

(1) 

»  *. 

(2) 

**. 

c. 

(*) 

I 

(1) 

k 

(2) 

702.  (*)  OPSEC .  OPSEC  requirements  have  been  considered  in 

developing  this  test  plan. 

a.  (*)  When  Force  test  and  evaluation  activities  are 
subject  to  monitoring  by  known  or  suspected  intelligence- 
collection  platforms,  the  following  types  of  information 
which  could  be  used  by  a  potential  enemy  should  not  be 
passed  by  uncovered  communications  or  otherwise  made  subject 
to  compromise: 

(1)  . 

(2)  . 


G 
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b.  (*)  Necessary  changes  to  run  schedules  and  plans 
caused  by  intruders  will  be  promulgated  by  the  OTD  as  fol¬ 
lows: 


(1) 

(2) 
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Annex  A 

Run  Plans  and  Geometries  (*) 

A101.  {*)  General 

An  Annex  may  be  used  to  provide  detailed  guidance  for 
executing  the  Teat  Plan *  such  ae  run  geometries . 

A  run  ie  an  exercise  involving  simulated  friendly  and 
threat  units*  and  associated  monitoring  and  instrumentation 
units*  conducted  to  acquire  data  pertinent  to  a  scenario. 

Run  plans  translate  scenarios  into  specific  events  and 
geometries*  and  provide  the  necessary  direction  to  all  test 
participants .  They  provide  the  required  start  events  ( e.g .* 
COMEX )*  the  movements  of  all  participants  ( course *  speed * 
depth  (or  altitude)  changes *  and  any  restrictions  to  them)* 
and  stop  events  (FINEX).  They  address  controlled  variables * 
as  shown  in  Table  A-l. 
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Table  A-l  (*) 

Illustration  of  a  Test-Variable  Matrix  (*) 


Additional  details  and  run  geometries  may  he  included. 


Run  plane  are  used  for  at-sea  tests  and  for  tests  at 
land-based  test  sites.  They  are  also  used  for  computer 
simulations  used  to  extend  the  data  base.  When  computer 
simulations  are  employedj  run  plans  for  validation  of  the 
simulation  should  be  included. 
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Annex  D 

Forms  and  Data  Sheets  (*) 

Provide  a  copy  of  each  non-standard  form 3  data  sheet , 
questionnaire t  etc.t  to  he  used  in  the  evaluation.  Include 
instructions  on  how t  whent  and  by  whom  these  are  to  he 
filled  out. 

Do  not  include  a  copy  of  any  standard  Navy  form  being 
used. 
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Annex  C 

Analytical  Methodology  (*) 

The  methodology  used  to  trace  the  path  from  data  collec- 
tion  through  reduction  to  arrive  at  numerical  results  for 
MOEs  and  MOSs  should  be  specified.  When  possible,  example 
calculations  shall  be  shown. 

To  assist  the  QTD  in  the  conduct  of  the  evaluation,  the 
criticality  of  various  inputs  should  be  addressed.  That  is, 
fall-back  positions/methodologies  should  be  explored  demon¬ 
strating  what  conclusions  may  be  obtained  in  the  absence  of 
various  data  points. 
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1006 .  Test  Plan  Approval  Authority 

a.  Deputy,  AIRTEVRON  Commanding  Officers,  or  cognizant 
ACOSs  are  authorized  to  approve  Test  Plans  when: 

(1)  The  Test  Plan  is  fully  consistent  with  a  CNO- 
approved  TEMP. 

(2)  The  Commander  has  not  indicated  a  desire  to 
review  the  Test  Plan  prior  to  its  approval. 

b.  Test  Plans  that  do  not  meet  these  criteria  are 
reviewed  by  the  Deputy  Chief  of  Staff  (Code  02),  the  Chief 
of  Staff  (Code  01),  or  the  Commander  (Code  00),  as  appro¬ 
priate. 

1007.  OPSEC  Requirements  of  Test  Plans 
a.  Background 

(1)  OPSEC,  as  it  relates  to  COMOPTEVFOR  testing,  may 
be  defined  as  the  identification  and  protection  of  a  broad 
spectrum  of  information  that  collectively  reveals  current 
and  future  U.S.  Military  capabilities,  plans,  and  opera¬ 
tional  procedures.  In  this  respect  it  encompasses  and 
relates  to  other  security  programs  such  as  SIGSEC  (signal 
security)  and  OPDEC  (operational  deception). 

(2)  Basic  guidance  on  OPSEC  is  contained  in  OPNAV- 
INST  3120.31,  CINCLANTFLTINST  C3100.10,  and  COMOPTEVFOR INST 
C3100.1. 


b.  Requirements  for  OPSEC  in  Test  Planning 

(1)  COMOPTEVFOR  testing  is  largely  devoted  to  veri¬ 
fying  the  capabilities  of  new  weapons  systems  and  developing 
tactics  for  their  use.  For  this  reason,  application  of 
OPSEC  thinking  to  OPTEVFOR  test  scenarios  is  extremely 
important,  to  avoid  unnecessary  disclosure  of  weapons  sys¬ 
tems  capabilities  and  limitations. 

(2)  The  application  of  OPSEC  thinking  to  OPTEVFOR 
test  scenarios  is  a  two-step  process: 

(a)  Identifying  those  elements  of  information 
that  require  protection  (e.  g.,  communications,  non-commu¬ 
nications  electromagnetic  emissions  and  tactics). 
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(b)  Establishing  means  of  protecting  these  elements 
during  OPTEVFOR  testing  and  during  the  subsequent  analysis  process. 
OPNAVINST  5510.143,  which  establishes  policy  on  SIGSEC,  and  OPNAVINST 
53490.2,  which  provides  guidance  on  cover  and  deception  planning, 
are  useful  for  this  purpose. 

(3)  OTDs  developing  Test  Plans  will  analyze  their  test 
programs  from  the  viewpoint  of  OPSEC,  and  include,  in  the  section 
on  security,  a  paragraph  stating  that  OPSEC  requirements  have  been 
considered.  (If  done  correctly,  questions  involving  SIGSEC  and 
the  possible  need  for  OPDEC  planning  will  be  addressed  as  well.) 

(4)  Test  scenarios,  the  interchange  of  information  during 
project  operations,  and  the  dissemination  of  test  data  will  be 
designed  to  minimize  availability  of  useful  information  to 
unauthorized  sources.  Necessary  instructions  will  be  included  in 
detailed  test  procedures. 

(5)  Prior  to  commencing  tests,  test  participants  will  be 
briefed  by  the  OTD,  or  his  representative,  on  security  requirements 
of  the  test. 

c.  Assistance  in  applying  OPSEC  requirements  to  individual 
Test  Plans  may  be  obtained  from  Force  Operations  and  Plans  (23) . 

1008.  Use  of  Photography  During  OT&E 

a.  Whenever  possible,  plan  to  make  use  of  photography 
(including  videotaping)  during  OT&E  to: 

(1)  Provide  illustrations  to  clarify  the  text  of  Evaluation 

Reports. 

(2)  Furnish  the  command  with  a  supply  of  OT&E  oriented 
(as  opposed  to  development-  or  sales-oriented)  illustrations  for 
use  in  briefings  and  presentations. 

b.  This  photographic  coverage  may  vary  from  amateur,  candid- 
type  photography  by  the  OTD  to  professional  coverage  by  Fleet 
Audio  Visual  Command  Atlantic.  Examples  of  types  of  photographic 
coverage  that  may  be  useful  in  Evaluation  Reports  or  in  briefings 
on  OT&E  are  as  follows: 

(1)  Photographs  of  test  personnel  using  handheld  equip¬ 
ment  (e.g.,  metal  detectors,  ordnance  examining/neutralization 
devices,  on-board  testers).  These  may  reduce  the  amount  of  text 
in  "Equipment  Description,"  and/or  may  provide  useful  illustrative 
vugraphs. 
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(2)  Photographs  of  equipment  displays  that  illustrate  points 
to  be  made  in  an  Evaluation  Report  or  briefing  (e.g.,  "before*'  and 
"after"  shots  of  scopes  that  illustrate  effects  of  electronic 
countermeasures,  shots  of  confusing  or  ambiguous  symbology). 

(3)  Photographs  of  damage  incurred  during  normal  operations 
that  illustrate  inherent  weaknesses  of  the  equipment  under  test 
(e.g.,  missile  fins  bent  during  normal  assembly,  handling,  or  load¬ 
ing  evolutions;  cracks  or  excessive  wear  incurred  during  routine 
use) . 


(4)  Photographs  of  the  test  system  underway  during  OT&E 
(e.g.,  SEAFOX  making  a  swimmer  recovery,  PEGASUS  at  high  speed  in 
heavy  seas,  the  F-18  flying  an  OT&E  mission).  These  may  be  used 

as  general  illustrations  in  reports  or  briefings,  or  may  illustrate 
specific  points  (e.g.,  heavy  spray  obscuring  a  gunner's  vision). 

(5)  Photographs  of  the  test  system  as  installed  in  the 
ship,  aircraft,  etc.,  for  general  information  or  to  illustrate 
an  important  aspect  of  the  installation  (e.g.,  inaccessibility 
for  maintenance,  antenna  blockage  by  superstructure). 

(6)  Motion  photography  (or  videotapes)  of  the  equipment 
in  operation,  for  general  information,  for  post-test  analysis,  or 
to  'illustrate  an  important  aspect  of  the  system  (e.g.,  CIWS 
engaging  a  target,  a  console  before  and  during  a  computer  hang-up). 

c.  When  OTDs/OTCs  have  obtained  photographs  of  OT&E,  they 
should  inform  the  Deputy  Chief  of  Staff  of  this  fact  in  an  informal 
memo  that  describes  briefly  what  is  available.  The  Deputy  Chief 
of  Staff  maintains  a  consolidated  file  of  this  information  for 
use  by  the  command.  Should  the  command  need  vugraphs  or  motion 
pictures  based  on  material  acquired  by  an  OTD/OTC,  tasking  is  to 
the  appropriate  Assistant  Chief  of  Staff  by  the  Deputy  Chief  of 
Staff. 


d.  Sources  of  Assistance  to  the  OTD/OTC 

(1)  The  Assistant  Chief  of  Staff  for  Operations  (Code  20) 
provides  scheduling  assistance  for  FLTAVCOMLANT  in  accordance 
with  CINCLANTFLT  Instruction  3150.1  (series). 

(2)  The  Comptroller  and  Force  Supply  Officer  (Code  014) 
advises  the  OTD/OTC  on  matters  associated  with  funding  require¬ 
ments  for  photographic  coverage,  including  film  and  processing 
costs . 


(3)  The  Director  of  Administration  (Code  10): 

(a)  Assists  the  OTD/OTC  in  completing  forms,  etc. 
associated  with  obtaining  photographic  services. 
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(b)  Makes  arrangements  for  OTDs/  OTCs  to  obtain  tem¬ 
porary  subcustody  of  cameras  charged  to  Graphic  Arts. 

1009.  Checklist.  The  attached  list  is  designed  to  help  the  OTD 
avoid  some  of:  the  more  frequent  errors  in  Test  Plans. 
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Test  Plan  Checklist 


R) 


1.  If  submarines  are  involved  in  any  way  (project  ships, 
sonar  targets,  etc.)  COMSUBPAC  and/or  COMSUBLANT  approval 

of  submarine  safety  aspects  is  requested.  (  ) 

2.  Objectives  and  evaluation  criteria  adequately  address 
all  elements  of  operational  effectiveness  and  operational 
suitability  pertinent  to  this  phase  of  OT&E.  (Note:  If 
the  objectives  contained  in  the  TEMP  are  wrong  or  incomplete 
—  don't  compound  the  problem  by  carrying  the  error  over 

into  the  Test  Plan!  Fix  them  now. )  (  ) 

3.  Limitations  to  scope  are  real  limitations  to  the 

evaluation  and  there's  no  way  to  eliminate  them.  (  ) 

4.  Equipment  description  is  as  concise  as  possible  — 
doesn't  repeat  the  switchology,  etc.,  contained  in  techni¬ 
cal  documentation  being  provided  test  operators  and  main¬ 


tenance  personnel.  (  ) 

5.  Testing  will  enable  accomplishment  of  objectives  and 

verifications  of  criteria.  (  ) 

6.  Testing  is  structured  to  provide  meaningful  data  for 

an  OPTEVFOR  Tactics  Guide.  (  ) 

7.  Where  applicable,  standardized  S-Tests  are  used.  (  ) 

8.  Data  requirements  and  responsibilities  for  data  col¬ 
lection  are  specified.  (  ) 

9.  Analysis  methods  are  specified  and,  where  necessary, 

definitions  (e.g,,  critical  failures,  no-tests,  etc.)  are 
provided.  (  ) 

10.  OPSEC  requirements  have  been  considered  (see  OPTEV- 

FORINST  C3100.1)  and  the  Test  Plan  so  states.  (  ) 

11.  Questionnaires  have  been  tailored  for  the  project  — 

i.e.,  they  ask  only  relevant  questions.  (  ) 


(Change  3) 


10-50 


Section  11 


Test  Operations 

1101.  OTP  Responsibilities  Before  Test  Operations  Begin 

a.  Draft  a  personal  letter  from  COMOPTEVFOR  to  the  commanding  (R 
officer  of  each  unit  scheduled  to  provide  key  services  during  the 
OT&E.  The  letter  should  introduce  the  purpose  and  objectives  of 
the  OT&E,  cite  the  time  frame  and  test  scenario  (area,  exercise, 
other  participants,  etc.),  provide  the  names  of  principal  OT&E 
project  personnel  (OTC,  OTD,  ship  riders),  and  request  the  under¬ 
standing  and  cooperation  of  the  commanding  officer.  While  there 
is  no  firm  "cookbook"  approach  to  such  a  personal  letter,  the 
sample  that  follows  illustrates  the  desired  approach. 

SAMPLE  LETTER  TO  CO,  RDT&E  SUPPORT  UNIT  (R 

* 

CAPT  Raymond  P.  Ilg,  USN  Date 

Commanding  Officer 
USS  NIMITZ  (CVN-68 ) 

FPO  New  York,  NY  09542 

Dear  Captain  Ilg, 

I  was  most  pleased  to  learn  that  your  fine  ship  has  been 
assigned  to  participate  in  the  OPEVAL  of  the  carrier-based 
Antisubmarine  Warfare  Module  (CV-ASWM).  My  purpose  in  writing 
to  you  is  to  pass  along  some  information  and  guidance  intended 
to  make  your  efforts  in  this  OPEVAL  more  effective. 

To  begin  with,  my  command  is  the  Navy's  sole  operational 
test  and  evaluation  activity.  I  report  directly  to  the  Chief 
of  Naval  Operations,  and  am  totally  independent  of  the  Navy's 
equipment  development  activities.  Briefly,  our  mission  is  to 
determine  -  based  upon  operational  testing  and  our  evaluation 
of  testing  results  -  whether  new  equipments  or  systems  should 
be  introduced  into  the  fleet.  To  accomplish  this  we  must: 

-  Measure  the  effectiveness  of  the  system/equipment  perform¬ 
ance  in  its  operational  environment.  "How  well  does  it  do  what 
it's  supposed  to  do  when  operated  by  fleet  personnel,  across  as 
wide  a  spectrum  of  sea  and  weather  conditions  as  we  can  encounter, 
in  the  face  of  as  accurate  a  representation  of  threat  and  work¬ 
load  as  we  can  generate?"  is  the  question  which  we  try  to  answer. 

-  Measure  the  suitability  of  the  system/equipment  in  its 
operational  environment.  "Is  it  reliable;  can  fleet  sailors 
maintain  it;  how  available  is  it;  is  it  logistically  supportable, 
is  it  compatible  with  other  systems  and  equipments,"  are  some 

of  the  questions  which  must  be  answered  here. 
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b.  As  the  date  to  begin  test  operations  approaches, 
check  to  make  sure  that: 

(1)  Appropriately  trained  personnel  will  be  available 
to  operate  and  maintain  the  equipment. 

(2)  The  equipment  to  be  evaluated  (including  special 
support  equipment)  will  be  installed  and  checked  out. 

(3)  Operator  and  maintenance  manuals,  the  ILSP,  and 
other  necessary  documentation  will  be  available  from  the  DA. 

(4)  Instrumentation  (including  range  instrumentation) 
will  be  available  and  in  working  order. 

(5)  Targets,  simulators,  electronic  warfare  services, 
etc.  will  be  available. 

(6)  Participants  have  received  required  test  directives 
(the  Test  Plan,  LOIs,  etc.),  and  understand  them. 

(7)  COMSUBLANT  and/or  COMSUBPAC  has  concurred  with 

the  safety  aspects  of  Test  Plans  that  involve  use  of  submarines 

(8)  RDT&E  support  services  remain  on  track. 

(9)  You  have  contingency  plans  for  the  unexpected. 

(10)  Arrangements  have  been  made  for  pre-test  briefings 
(including  arrangements  for  additional  briefers,  if  necessary). 

(11)  Special  data  forms  and  questionnaires  are  available 
in  sufficient  quantity. 

(12)  If  appropriate,  rehearsals  of  test  operations 
are  scheduled.  (Rehearsals  are  good  if  they  increase  the 
likelihood  of  obtaining  meaningful  data.  Do  rehearse  data 
collection.  Rehearsals  are  bad  if  they  destroy  operational 
realism.  Don't  eliminate  the  possibility  of  having  unalerted 
operators ,  etc . ) 

(13)  Pre- faulted  modules  (for  example)  will  be  available 
for  a  maintenance  demonstration,  if  one  becomes  necessary. 

c.  Immediately  prior  to  the  start  of  test  operations, 
make  sure  that: 

(1)  All  hands  know  what  they're  supposed  to  do. 


(2)  The  equipment  to  be  evaluated  is  in  working 

order . 


(3)  Other  equipment  necessary  to  the  test  scenario, 
and  instrumentation  equipment,  are  in  working  order. 

(4)  Personnel  to  activate/deactivate  data  recorders, 
and  back-up  data  takers,  are  in  place. 

(5)  As  necessary,  time  synchronization  and  communica¬ 
tions  have  been  established. 

(6)  Data  forms  have  been  distributed,  as  necessary. 

(7)  Contingency  plans  have  been  discussed  with 
appropriate  personnel  (e.g.,  with  the  Commanding  Officer  of 
the  test  ship ) . 

1102.  OTP  Responsibilities  During  Test  Operations.  Ensure 


a.  Tests  are  conducted  in  accordance  with  test  directives 
any  deviations  are  noted,  their  impact  is  assessed,  and 
necessary  corrective  action  is  taken.  Contingency  plans  are 
implemented,  as  necessary. 

b.  Data  recorders  are  refilled  as  necessary;  recorded 
data  are  stored  in  a  safe  place. 

c.  Unusual  events  during  testing  that  may  have  some 
effect  on  test  results  are  noted  (e.g.,  in  the  OTD  Journal). 

d.  Data  forms  are  completed  as  specified  in  the  Test 
Plan. 

e.  Reports  are  generated  as  specified  in  the  Test  Plan, 
and  as  discussed  in  paragraph  1105. 

1103.  OTD  Responsibilities  After  Test  Operations.  Ensure 
that: 

a.  Questionnaires  are  distributed,  filled  in,  and 
returned  to  the  OTD  (or  as  specified  in  the  Test  Plan). 

b.  Necessary  debriefs  are  conducted,  as  are  post- test 
interviews . 

c.  All  other  data  are  delivered  to  the  OTD  (or  as 
specified  in  the  Test  Plan). 
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d.  When  necessary,  a  maintenance  demonstration  is 
conducted. 

e.  Analysis  proceeds  as  necessary  to  allow  the  Evaluation 
Report  deadline  to  be  met. 

1104.  Release  of  Test  Data.  In  general,  raw  OT&E  test  data 
(particularly  failure  data)  can  be  made  available  to  the  DA 
immediately.  However,  release  of  these  data  must  be  accompanied 
by  safeguards  to  ensure  that: 

+ 

a.  Status/progress  reports  within  the  Navy  on  OT&E  test 
results  are  made  only  by  COMOPTEVFOR  or  his  designated 
representative . 

b.  Information  for  higher  authority  on  OT&E  test  results 
is  prepared  only  by  COMOPTEVFOR . 

c.  No  press  releases  of  any  kind  are  made  based  on  OT&E 
test  data . 

d.  Contractors  are  enjoined  from  making  use  of  OT&E 
test  data  in  advertising  or  selling  weapons  systems. 

1105.  Reports  Associated  With  Test  Operations 

* 

a.  Commencement  of  OPEVAL.  When  am  OPEVAL  starts  the 
cognizant  ACOS  is  required  to tramsmit  a  message  from  COMOP¬ 
TEVFOR  to  CNO  stating  that  "CNO  Project  XX-OT-II  (OPEVAL) 

on  the  (equipment  naune)  commenced  (DTG( local) ). "  Comments, 
particularly  unanticipated  limitations,  may  be  included  in 
this  message.  It  is  an  OTC/OTD  responsibility  to  draft  this 
message  for  the  ACOS. 

b.  Deficiency  Report.  A  deficiency  report  is  submitted 
to  COMOPTEVFOR  by  the  prosecuting  agency  (e.g.,  VX  Squadron, 
project  ship)  when  a  project  is  being  delayed  because  the 
equipment  cannot  be  operated,  because  required  support  is 
lacking,  or  because  of  prolonged  delay  in  equipment  delivery. 
These  reports  are  by  letter,  speed  letter,  or  message.  In 
the  case  of  a  project  ship  with  the  OTD  embarked,  the  OTD 
drafts  the  deficiency  report  and  prefaces  it  with  "OTD 
Sends."  COMOPTEVFOR  may  in  turn  send  a  deficiency  report  to 
the  CNO,  with  an  information  copy  to  the  cognizant  systems 
command,  CHNAVMAT,  and  the  prosecuting  agency.  Deficiency 
reports  will  contain  a  summary  of  the  deficiency,  action 
taken,  and  recommended  action. 
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Section  12 


The  Evaluation  Report 


1201.  Introduction 


a.  There  are  two  products  of  OT&E:  the  Tactics  Guide, 
in  which  OPTEVFOR  addresses  how  to  use  a  system,  and  the 
Evaluation  Report.  An  Evaluation  Report  provides  the  CNO 
with  COMOPTEVFOR ' s  conclusions  regarding  a  system's  operational 
effectiveness  and  operational  suitability,  and  his  recommenda¬ 
tions  regarding  the  system  (further  development,  procurement 
and  production,  additional  T&E,  etc.).  In  addition,  an 
Evaluation  Report  provides  the  information  (test  results, 
evaluation  criteria,  etc.)  to  substantiate  the  conclusions 

and  recommendations. 

b.  In  high-interest  programs,  COMOPTEVFOR  often  provides 
his  conclusions  and  recommendations  to  the  CNO  before  formal 
Evaluation  Reports  are  issued  —  in  messages,  in  briefings 
associated  with  CEB,  DNSARC,  or  DSARC  meetings,  etc.  In 
these  cases,  major  milestone  decisions  are  sometimes  made 
before  formal  Evaluation  Reports  are  issued.  This  circumstance 
does  not  alter  the  requirement  for  the  report  —  its  record 

of  OT&E  is  still  required,  and  odds  are  that  sooner  or  later 
it  will  be  used.  A  few  examples  of  how  the  report  might  be 
used: 


(1)  Problems  with  newer  systems  in  the  fleet  can 
cause  T&E  reports  to  be  examined  for  clues  to  the  sources  of 
the  problems  —installation  differences,  design  changes 
incorporated  since  testing,  etc. 

(2)  Evaluation  Reports  have  been  a  major  data  source 
for  recent  GAO  investigations  of  Navy  RDT&E. 

Evaluation  Reports  are  never  "OBE"  because  program  decisions 
have  been  made  prior  to  their  publication. 

1202 .  Types  of  Evaluation  Reports  There  are  two  categories 
of  Evaluation  Reports:  formal  reports  that  are  permanent 
records  of  OT&E,  and  quick-look  reports  that  are  temporary 
substitutes  for  formal  reports. 

a.  Formal  Evaluation  Reports  usually  consist  of  letters 
signed  by  the  Commander,  accompanied  by  enclosures.  The 
letters  are  addressed  to  the  CNO,  and  are  written  Admiral - 
to-Admiral,  emphasizing  system  operational  effectiveness  and 
operational  suitability  and  the  program  decision  (e.g.. 
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full-scale  development,  ASU)  under  consideration.  Enclosures 
are  written  primarily  for  the  DA,  and  emphasize  the  details 
of  test  and  analysis  and  detailed  changes/corrective  actions 
(not  in  themselves  of  interest  to  the  decision  maker)  that 
are  recommended.  There  are  three  types  of  formal  Evaluation 
Reports,  as  follows: 

(1)  A  Report  covers  a  complete  phase  of  OT&E  (e.g., 
OT-IIIA)  in  a  single  document. 

(2)  A  Partial  Report  covers  part  of  a  phase  of  OT&E. 

It  is  is  used  when  a  phase  takes  a  long  time  to  complete  — 
in  order  to  keep  CNO  aware  of  OT&E  progress.  Some  examples 
of  phases  in  which  Partial  Reports  were  used: 

(a)  The  F- 14A/PHOENX X  OPEVAL  was  a  time-consuming 
T&E  effort  that  exercised  the  F-14A/PH0ENIX  sequentially  in 
increasingly  difficult  mission  areas.  Ten  Partial  Reports  were 
issued,  one  after  completion  of  testing  in  each  mission  area. 

(b)  F-14A/PH0ENIX  FOT&E  involved  reliability  tests 
of  production  AIM-54  missiles  during  deployments  of  four  dif¬ 
ferent  aircraft  carriers.  Because  of  the  long  time  between  the 
beginning  of  the  first  and  completion  of  the  fourth  deployment, 
results  of  earlier  deployments  were  published  in  Partial  Reports. 

(3)  A  Summary  Report  is  prepared  when  it  is  necessary 
to  integrate  information  from  a  series  of  Partial  Reports  in 
order  to  make  overall  system-level  conclusions  and  recommenda¬ 
tions.  Summary  Reports  that  have  been  published  (to  date 
they  have  been  rare)  have  been  letters  without  enclosures; 
they  made  references  to  the  Partial  Reports  for  details, 
eliminating  the  need  for  enclosures. 

b.  A  Quick-look  Report  is  a  temporary  substitute  for  a 
formal  Evaluation  Report.  Usually  it  covers  an  entire  phase 
of  OT&E,  and  substitutes  temporarily  for  a  Report.  It  is 
usually  in  message  format,  is  addressed  to  the  CNO,  and  has 
essentially  the  same  emphasis  as  the  letter  portion  of  a 
Report.  Differences  between  Quick- look  Reports  and  Reports 
are  as  follows: 

(1)  A  Quick-look  Report  is  not  backed  up  by  the 
substantiating  detail  contained  in  the  enclosure  to  a  Report. 

(2)  Quick-look  Report  results,  conclusions,  and 
recommendations  may  be  subject  to  modification  because  they 
are  based  on  incomplete  analysis. 


(3)  A  Quick-look  Report  may  defer  non-critical 
aspects  of  the  evaluation  to  the  formal  Report. 

(4)  A  Quick-look  Report  may  contain  more  detail  in 
some  areas  than  the  Report  letter.  Usually,  this  detail  is 
associated  with  recommended  system  changes  not  in  themselves 
of  interest  to  the  decision  maker,  but  that  COMOPTEVFOR 
wants  the  DA  to  know  about  immediately. 

c.  As  already  indicated,  a  Report  or  a  Partial  Report 
is  usually  a  letter  backed  up  by  an  enclosure;  a  Quick-look 
Report  is  usually  a  message.  Variations  from  the  usual  have 
been  as  follows: 

(1)  Letters  Without  Enclosures.  These  have  included 
reports  on  IOT&E  that  was  restricted  to  operational  evalua¬ 
tion  of  DT&E  results  (i.e.,  no  operational  testing).  The 
appropriate  DT&E  reports  were  referred  to  for  detail,  in 
lieu  of  enclosures. 

(2)  Formal  Reports  in  Message  Format.  From  time  to 
time,  COMOPTEVFOR  has  been  asked  to  evaluate  a  piece  of 
equipment  not  covered  by  a  TEMP  —  something  a  contractor  or 
a  Navy  lab  has  developed  that  is  worth  checking  out  from  an 
operational  viewpoint.  No  T&E  funds  are  involved,  and  sim¬ 
ple  tests  during  an  ongoing  OPEVAL  are  all  that's  required. 
In  cases  like  this,  formal  Reports  in  message  format  have 
been  issued,  providing  the  DA  with  an  evaluation  as  quickly 
as  possible  without  the  wide  dissemination  of  information 
required  in  projects  covered  by  TEMPs. 

(3)  Quick-look  Reports  in  Letter  Format.  Occasion¬ 
ally,  the  CNO-approved  evaluation  criteria  have  been  so  de¬ 
tailed  that  the  simplest  way  to  address  them  (a  table  com¬ 
paring  results  with  criteria)  is  too  complex  for  an  under¬ 
standable  message.  In  these  cases.  Quick-look  Reports  have 
been  prepared  in  letter  format  and  hand-carried  to  CNO. 


1203.  When  Are  Evaluation  Reports  Required? 


a.  Evaluation  Reports  are  required  as  specified  in  Part 
II  of  the  TEMP,  generally  at  the  completion  of  a  phase  of 
OT&E.  Reports  not  required  by  the  TEMP: 


(1)  May  be  requested  by  agencies  outside  OPTEVFOR. 
Note  that  according  to  OPNAVINST  3960.10,  such  requests  must 
be  approved  by  CNO  (OP-098)  before  you're  required  to  honor 
them. 
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(2)  May  be  directed  by  COMOPTEVFOR.  (Partial  Reports 
are  often  of  this  variety. ) 

b.  Publication  deadlines  for  Evaluation  Reports  are 
specified  in  COMOPTEVFOR INST  3960. 2A. 

c.  Quick-look  Reports  are  temporary  documents  —  they 
are  always  superseded  by  formal  Evaluation  Reports. 

1204.  The  Logical  Organization  of  an  Evaluation  Report 

a.  According  to  Annex  A  to  this  Guide,  the  fundamental 
requirement  for  good  writing  is  logical  organization  — 
making  the  discussion  proceed  in  logical  steps  from  beginning 
to  end,  without  irrelevancies  or  digressions.  The  standard 
format  for  the  letter  portion  of  a  formal  Evaluation  Report 
(which  is  basically  the  same  for  a  Quick-look  Report)  is 
designed  to  provide  a  logical  organization  in  eight  or  nine 
paragraphs.  The  logical  flow  of  information  in  these  paragraphs 
is  illustrated  in  Figure  12-1.  In  Figure  12-1,  the  heavy 
arrows  that  lead  from  box  to  box  represent  the  flow  of 
information.  The  three  boxes  on  the  left,  representing 
paragraphs  1  through  4  of  the  letter,  provide  the  essential 
background  information  —  the  "why"  and  "what"  of  the  OT&E. 

The  box  in  the  center  (paragraph  5)  is  the  "how"  of  testing. 

The  boxes  on  the  right  (paragraphs  6  through  9)  provide  the 
meat  of  the  evaluation  —  the  major  results  of  testing, 
operational  factors  that  influence  interpretation  of  the 
results,  and  COMOPTEVFOR ' s  conclusions  and  recommendations. 

The  thin  horizontal  arrows  between  boxes  on  the  left  and 
boxes  on  the  right  illustrate  where  "questions  asked"  on  the 
left  are  "answered"  on  the  right.  More  on  this  below. 

b.  Paragraph  1,  Purpose,  introduces  the  report  by 
stating  the  reason  for  the  phase  of  OT&E  (i.e.,  the  program 
decision  under  consideration),  and  the  basis  for  the  evaluation 
(i.e.,  an  investigation  of  operational  effectiveness  and 
operational  suitability). 

c.  Paragraph  2,  Equipment  Description  (or  System 
Description  —  your  choice)  is  a  short  description  of  what 
was  tested.  It  emphasizes  the  function  of  the  equipment, 
and  significant  difference  between  what  was  tested  and  the 
proposed  operational  configuration . 

d.  Paragraph  3,  Background,  briefly  summarizes  the 
reason  the  equipment  is  being  developed  and  the  T&E  conducted 
before  the  phase  of  OT&E  being  reported. 
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e.  Paragraph  4,  Scope ,  has  three  standard  subparagraphs: 

(1)  Objectives.  These  define  the  elements  of  opera¬ 
tional  effectiveness  and  operational  suitability,  as  shown 
in  Figure  12-1. 

(2)  Evaluation  Criteria.  These  quantify  the  objectives. 
They  are  theMOEs  and  MOSs. 

(3)  Limitations  to  Scope.  These  are  the  ways  in 
which  COMOPTEVFOR ' s  evaluation  is  limited.  They  identify 
objectives  (or  portions  of  objectives,  MOEs,  or  MOSs)  that 
could  not  be  fully  assessed. 

f.  Paragraph  5,  Project  Operations,  is  a  brief  narrative 
that  describes  how  testing  was  conducted  (the  scenarios  and 
the  magnitude  of  testing  —  how  many  bombs  were  dropped,  how 
long  the  equipment  was  operated,  etc.).  It  gives  the  reader 
an  idea  of  the  data  base. 

g.  Paragraph  6,  Results,  presents  the  major  results  of 
test  and  analysis  —  keyed  to  the  objectives,  as  indicated 
by  the  thin  horizontal  arrows  in  Figure  12-1.  All  objectives 
and  all  the  evaluation  criteria  associated  with  them  are 
addressed  in  this  paragraph,  except  those  specifically 
excluded  by  Limitations  to  Scope. 

h.  In  most  OT&E,  once  the  results  have  been  presented, 
the  complete  logic  for  conclusions  and  recommendations  has 
been  established.  In  some  cases,  however,  operational 
reasoning  suggests  conclusions  and/or  recommendations  that 

do  not  derive  directly  from  results.  Paragraph  7,  Operational 
Considerations ,  is  an  optional  paragraph  that  is  used  to 
develop  this  operational  reasoning.  Some  examples: 

(1)  In  testing,  the  following  results  were  obtained: 

(a)  MTBF:  120  hours  (criterion:  150  hours). 

(b)  MTTR:  2  minutes  (criterion:  <_  60  minutes). 

(c)  AQ:  99.9%  (criterion:  >_  98%). 

A  direct  conclusion  from  these  results  would  be  that  the 
system  was  not  operationally  suitable  because  the  system  did 
not  meet  the  reliability  criterion.  COMOPTEVFOR,  however, 
felt  that: 
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(a)  The  system  was  "up"  most  of  the  time,  as 
evidenced  by  the  fact  that  hQ  was  high. 

(b)  The  high  A  was  attributable  to  very  short 
repair  times,  evidenced  in°a  very  low  MTTR. 

(c)  With  the  very  low  MTTR,  an  MTBF  of  120  hours 
was  acceptable  from  an  operational  viewpoint. 

COMOPTEVFOR' s  views  were  developed  in  Operational  Considera¬ 
tions;  it  provided  the  rationale  for  a  conclusion  that  the 
system  was  operationally  suitable,  even  though  it  did  hot 
meet  the  reliability  criterion. 

(2)  During  OPEVAL  of  System  X,  repeated  failures  of 
service-approved  System  Y  were  observed.  System  Y  was  being 
used  as  a  backup  data  collection  device,  and  its  failures 

had  no  adverse  effect  on  the  evaluation  of  System  X.  Therefore 
System  Y's  failures  would  not  be  discussed  under  Limitations 
to  Scope.  Nor  would  they  be  discussed  under  Results;  deter¬ 
mining  System  Y's  reliability  was  not  an  objective  of  the 
System  X  OPEVAL.  But  COMOPTEVFOR  desired  to  bring  a  potential 
System  Y  reliability  problem  to  the  CNO's  attention.  Opera¬ 
tional  Considerations  was  used  to  report  the  observed  failures, 
and  substantiated  a  recommendation  to  investigate  System  Y's 
reliability  in  the  fleet. 

(3)  During  OPEVAL  of  an  acoustic  signal  processor, 
the  system  met  all  the  evaluation  criteria.  During  project 
operations,  operators  in  the  Project  Ship  pointed  out  an 
apparently  simple  change  in  processor  logic  that  could 
provide  a  significant  increase  in  capability  —  allowing 
target  localization  in  addition  to  the  designed  capability 
of  providing  target  bearing.  COMOPTEVFOR  discussed  this 
possibility  in  Operational  Considerations,  and  then  concluded 
(based  on  test  results)  that  the  processor  was  operationally 
effective  and  operationally  suitable.  COMOPTEVFOR ' s  first 
recommendation,  however,  was  not  for  ASU,  the  usual  OPEVAL 
recommendation  on  an  operationally  effective  and  operationally 
suitable  system.  Instead,  COMOPTEVFOR  recommended  that  the 
feasibility  of  providing  a  target  localization  capability  be  . 
considered,  then  ASU  and  production. 

i.  In  paragraph  8,  Conclusions,  COMOPTEVFOR  answers  the 
fundamental  questions  implied  in  paragraph  1,  as  is  indicated 
by  the  thin  horizontal  arrows  toward  to  top  of  Figure  12-1; 
he  provides  his  conclusions  on  operational  effectiveness  and 
operational  suitability. 
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j.  In  paragraph  9,  Recommendations ,  COMOPTEVFOR  addresses 
the  reason  for  this  OT&E  —  the  up-coming  program  decision. 

I f  COMOPTEVFOR  recommends  against  the  equipment  ( e .  g . , 
against  ASU),  he  recommends  what  should  be  done  instead 
(e.g.,  program  cancellation;  fix  and  retest). 

k.  The  enclosure  to  the  letter  provides  substantiating 
detail  (primarily  in  results)  and  additional  recommendations 
of  interest  primarily  to  the  DA. 

1205.  Fundamentals  of  Writing  an  Evaluation  Report 

a.  Be  familiar  with  this  Guide  —  particularly  with 
paragraph  1206. 

b.  Read  recent  reports  —  particularly  any  on  similar 
systems  —  to  get  a  feel  for  how  it's  done. 

c.  Write  the  enclosure  first.  A  lot  of  it  (equipment 
description,  background,  test  procedures,  methods  of  analysis, 
etc. )  can  be  lifted  out  of  the  Test  Plan.  The  main  effort 

is  adding  the  results  to  the  various  E- tests  and  S-tests. 

d.  When  that  has  been  done  —  sit  back  and  think  what 
it  means.  Do  the  results  indicate  operational  effectiveness 
and  operational  suitability?  What  are  you  going  to  propose 
as  recommendations?  You  must  have  these  in  your  head  before 
you  go  on  to  the  next  step. 

e.  Write  the  letter. 

(1)  Make  it  Admiral-to-Admiral .  And  assume  the 
Admiral  on  the  receiving  end  has  only  general  familiarity 
with  the  warfare  area  the  report  covers  —  if  it's  a  report 
on  aircraft,  assume  the  Admiral  reading  it  came  out  of 
destroyers.  If  you  make  this  assumption  —  and  remember 

it  while  you're  writing  —  you'll  avoid  trade  jargon  and 
the  alphabet  soup  of  too  many  acronyms. 

(2)  Concentrate  on  the  logic  of  what  you're  writing. 
Key  the  results  to  objectives,  ana  make  sure  results  sub¬ 
stantiate  conclusions  and  make  recommendations  obvious.  Make 
sure  results  are  results  —  "The  radar  was  not  effective  in 
AAW"  is  a  conclusion;  "The  radar  detected  only  two  targets 

in  117  valid  detection  opportunities"  is  a  result. 

f.  Let  the  letter  sit  for  a  day  or  two.  Then  read  it 
for  clarity.  Fix  what  needs  fixing.  Then  test  the  letter 
using  the  Letter  Report  Checklist  of  paragraph  1208.  Fix 
what  needs  fixing.  Then  edit  the  letter  for  conciseness. 
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g.  Go  back  to  the  enclosure.  Eliminate  duplication  in 
Equipment  Description,  Background,  etc.  using  "See  basic 
letter."  Test  the  enclosure  against  the  Enclosure  Checklist 
of  paragraph  1208.  Fix  what  needs  fixing.  Edit  for  con¬ 
ciseness. 

1206.  Format  and  Contents  of  Evaluation  Reports.  The  for¬ 
mat  of  Reports,  Partial  Reports,  and  Summary  Reports  is 
illustrated  in  the  following  pages,  along  with  guidance 
regarding  content  and  desired  level  of  detail.  In  the 
example  which  follows,  standard  headings  and  samples  of 
content  are  presented  in  normal  type.  Explanatory  comments 
are  in  italics.  Keep  in  mind  that  the  example  is  a  guide  — 
it’s  not  holy  —  but  don’t  change  format  unless  it  improves 
the  report. 
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CLASSIFICATION* 

From:  Commander  Operational  Test  and  Evaluation  Force 
To:  Chief  of  Naval  Operations 

Subj :  Initial  Operational  Evaluation  of  the  New  Weapons 
System  (OPNAV  Report  Symbol  3960-12)  (*) 

For  an  OPEVAL  report,  delete  the  word  "Initial . " 

For  OT-III  or  OT-IV,  replace  " Initial "  with  " Follow-on ." 

Ref:  (a)  (Secret)  COMOPTEVFOR  XXXXXXZ  Mar  1977 

Keep  the  references  in  the  letter  to  an  absolute 
minimum.  References  are  lettered  consecutively  in 
the  order  in  which  they  appear  in  the  text. 

Enel:  (1)  (Classification)  CNO  Project  999-OT-I  Report 

Details  (*) 

1.  (*)  Purpose .  This  report  provides  COMOPTEVFOR ' s  initial 

operational  evaluation  of  the  New  Weapons  System,  performed 
under  CNO  Project  999-OT-I.  The  purpose  of  the  evaluation 
was  to  assess  the  potential  operational  effectiveness  and 
operational  suitability  of  the  New  Weapons  System,  and  its 
readiness  for  full-scale  development.  The  evaluation  was 
based  on  results  of  operational  tests  conducted  under  Pro¬ 
ject  999-OT-I,  supplemented  by  results  of  DT-I  and  opera¬ 
tional  experience.  This  report  cancels  and  supersedes 
reference  (a) ,  the  Quick-look  Report. 

This  is  the  form  of  the  paragraph.  The  author  must  adjust 
it  as  necessary  for  accuracy. 

1.  For  an  OPEVAL  report,  "to  assess  the  potential" 
becomes  "to  determine ;"  "full-scale  development"  becomes 
" approval  for  service  use  and  production.  " 

DOWNGRADING  STATEMENT  If  required  and  not  on  cover  sheet. 

CLASSIFICATION* 

*If  applicable,  insert  appropriate  classification.  Do  not 
use  on  UNCLASSIFIED  reports. 
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2.  For  FOT&E ,  replace  the  second  sentence  with  " The 
purpose  of  the  evaluation  was  to  determine  the  operational 
effectiveness  and  operational  suitability  of  the  production 
configuration  of  the  New  Weapon  System t  and  its  readiness 
for  full-rate  production.  " 

3.  For  a  Summary  Report t  change  the  lead-in  to  read 

" This  summary  report  provides...."  There  probably  won't  be 
a  Quick-look  to  ref erence t  but  each  Partial  Report  should 
be  referenced. 

4.  For  a  Partial  Report t  use  a  Purpose  paragraph  with 
the  following  form: 

1.  (*)  Purpose.  This  reports  the  fifth  phase  of  COMOPTEVFOR ' s 
initial  operational  evaluation  of  the  New  Weapons  System, 
performed  under  CNO  Project  999-OT-I.  The  purpose  of  the 
overall  evaluation  is  to  assess  the  potential  operational 
effectiveness  and  operational  suitability  of  the  New  Weapons 
System,  and  its  readiness  for  full-scale  development.  The 
fifth  phase  of  the  evaluation  concentrated  on  New  Weapons 
System  performance  in  the  presence  of  target  decoys  and 
chaff.  The  evaluation  was  based  on  results  of  operational 
tests  conducted  under  Project  999-OT-I,  supplemented  by 
operational  experience. 

2.  (*)  Equipment  Description.  The  New  Weapons  System  is 

a  ..... .  .T.  .  designed  to"  provide  surface  ships  with  a 

capability  to  detect,  track,  and  destroy  . 

Major  components  include . . .  Details  are  pro¬ 

vided  in  enclosure  (1) . 

This  paragraph  may  be  titled  System  Description  if  more 
appropriate  to  the  test  item. 

This  paragraph  provides  a  brief  statement  of  the  important 
functions /characteristics  of  tne  equipment  or  system  which 
was  tested.  View  this  statement  as  a  reminder  to  3-star 
readers  who  have  already  been  exposed  to  the  equipment. 

3.  (*)  Background .  The  New  Weapons  System  is  being  developed 

to  counter  the  projected  .  threat.  Development 

testing  (DT-I)  was  conducted  at  a  land-*  fesed  test  site  and 

in  USS  .  in  1976.  While  DT-I  results  were 

generally  satisfactory,  deficiencies  were  identified  in  the 

system's  capability  to  .  As  a  result,  the  system 

was  modified  to  include  a .  for  OT-I. 
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This  paragraph  summarizes  in  a  concise  narrative  the  major 
events  which  led  to  the  testing  being  reported.  As  appro - 
priate  to  the  particular  report ,  this  paragraph  includes  the 
requirement  for  the  equipment ,  deficiencies  or  inadequacies 
identified  in  previous  testing  or  in  operation ,  and  major 
development  milestones . 

4.  (*)  Scope 

a.  (*)  Objectives.  The  objectives  of  Project  999-OT-I 

were  to:  ~~ 

(1)  (*)  Determine  that  the  .  can  detect  . 

at  operationally  useful  ranges. 

(2)  (*)  Determine  the  capability  of  the  system  to 

track  .  in  clear  and  countermeasures  environments. 

(3;  (*)  Determine  the  potential  target  kill  proba¬ 
bility  of  the  . 

(4)  {*)  Assess  the  potential  reliability,  maintain¬ 
ability,  and  availability  of  the  system. 

(5)  (*)  Assess  the  potential  of  the  .  to  be 

supported  logistically  in  the  fleet. 

(6)  (*)  Assess  the  potential  compatibility  and  inter¬ 
operability  of  the  . 

(7)  (*)  Make  a  preliminary  assessment  of  training 
planned  for  fleet  operators  and  maintenance  personnel. 

These  are  the  objectives  as  stated  in  the  Test  Plan.  In  a 
Partial  Report,  the  objectives  will  be  those  of  the  particu¬ 
lar  phase  being  reported.  In  a  Summary  Report,  or  in  a 
Report,  they  will  be  the  complete  objectives  of  the  phase 
( e.g .,  OT-I). 

i 

b.  (*)  Evaluation  Criteria.  CNO  specified  that  the  New 
Weapons  system  meet  the  following  criteria  in  OT-I: 

(1)  (*)  Detection  Range:  XXX  m  (YYY  yards). 

(2)  (*)  Tracking  Capability:  X  targets  simultaneously. 

(3)  {*)  . 
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Include  here  the  appropriate  thresholds  for  the  required 
operational  characteristics  of  the  TEMP. 

Note  that  each  criterion  listed  here  is  addressed  either  in 
Limitations  to  Scope  or  in  Results  in  the  letter .  If  the 
actual  criteria  are  in  such  detail  as  to  he  inappropriate 
to  report  to  decision-makers  on  (e.g.,  detailed  environmental 
criteria  for  safe  storage  of  ordnance) ,  then  summarize  them 

here  ( e.g .,  adequate  environmental  control  for  . ), 

report  results  at  that  level ,  and  put  the  details  in  the 
enclosure. 

And  don't  include  criteria  which  simply  repeat  objectives  — 
they  should  amplify  objectives,  usually  with  numbers. 

c.  (*)  Limitations  to  Scope.  List  here  the  limitations 
to  the  evaluation  which  tend  to  add  qualifiers  to  the  results , 
conclusions ,  and  recommendations.  These  limitations  include 
those  predicted  in  the  Test  Plan ,  and  those  imposed  by 
unpredicted  circumstances  encountered  during  testing. 

Limitations  should  be  expressed  so  that  their  import  is 
readily  understood ,  e.g.,  "Since  testing  was  limited  to  the 
tropics,  system  performance  in  cold  climates  could  not  be 
evaluated. "  Keep  in  mind  that  these  represent  limitations 
to  the  evaluation  after  it's  all  over.  They  have  nothing  to 
do  with  how  hard  it  was  to  get  services,  or  how  long  it 
took.  If  the  job  of  evaluating  got  done,  there  are  no 
limitations . 

5.  (*)  Project  Operations.  Project  operations  were  con¬ 
ducted  in  USS  . .  from  13  November  to  20  December 

1977.  The  New  Weapons  System  was  exercised  in  simulated 

.  engagements  against  single  and  multiple  . 

Twenty  engagements  were  conducted,  10  with  four  simultaneous 

targets;  XX  rounds  of . were  expended.  All  targets 

employed  ECM  (electronic  countermeasures) . 

This  paragraph  is  a  brief  narrative  description  of  what  was 
done  to  accomplish  the  objectives.  It  includes  such  signifi¬ 
cant  information  as  number  of  firings,  total  hours  of 
equipment  operation ,  etc.,  and  identifies  the  maj or  fleet 
service (s )  provided.  Details  of  services  provided  are  con¬ 
tained  in  the  enclosure. 
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6.  (*)  Results.  Details  of  tests  and  results  are  contained 
in  enclosure  (1) .  Major  test  results  are  listed  below: 

Reeulte  are  presented  in  the  same  order  as  the  objectives. 
They  address  all  objectives  (and  associated  evaluation 
criteria)  unless  Limitations  to  Scope  excludes  one  or  more. 

a.  (*)  Detection  Capability.  The  capability  of  the  New 
Weapons  System  to  detect  threats  of  interest  is  summarized 
in  Table  1. 

Table  1  (*) 

Detection  Capability  (*) 

b.  (*)  Tracking  Capability . 

c.  (*)  Target  Kill  Capability . 

d.  (*)  Reliability . 

e .  ( * )  Maintainability . 

f.  (*)  Availability . 

g.  (*)  Logistics  Supportability . 

h.  (*)  Compatibility . 

i .  ( * )  Interoperability . 

j.  (*)  Training . . . 

7.  (*)  Conclusions 


a.  (*)  The  New  Weapons  System  has  the  potential  to  be 

operationally  effective,  based  on  its  demonstrated  ability 
to  destroy  . 

b.  (*)  The  New  Weapons  System  has  the  potential  to  be 

operationally  suitable,  provided  that  . 

Normally ,  in  a  well-written  report ,  conclusions  follow 
directly  from  test  results.  When  this  is  not  the  case 3 
precede  the  Conclusions  paragraph  with  an  Operational 
Considerations  paragraph ,  as  discussed  in  paragraph  1204.h. 
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a.  (*)  Approve  the  New  Weapons  System  for  full-scale  develop¬ 
ment. 

b.  (*)  Incorporate  the  following  changes  into  the  New  Weapons 
System: 

Kc.cofid.inQ  to  VOV  Directive  5000.  3,  one  of  the  purposes  of  OTlsE  is 
to  identify  the.  need  for  any  modifications .  It  is  pfiopefi,  the.fie.fo fie, 
to  make  fiecommendations  for  hardware  changes  that  will  increase 
operational  effectiveness  and/or  operational  suitability.  However, 
care  must  be  taken  to  avoid  usurping  the  DA 's  responsibility .  A void 
recommending  a  specific  modification  such  as  adding  a  10-ohm  resistor 
there  may  be  a  better  way  to  accomplish  the  fix.  If  reference  to 
a  10-ohm  resistor  will  help  the  VA  understand  the  nature  of  the 
required  modification,  recommend  a  modification  " such  as  a  10-ohm 

resistor  in  . "  Also  avoid  usurping  the  decision-making 

authority' s  res  pons ibility  to  consider  cost  tradeo  ffs .  In  the  case 
where  a  modification  coula  provide  a  capability  not  designed  into 
the  equipment,  do  not  recommend  its  incorporation.  Rather,  recommend 
"Consider  incorporating . " 

It  is  proper  to  make  recommendations  regarding  the  need  for  future 
testing  (F 0T6E,  for  example). 

c.  (*)  Incorporate  the  additional  recommendations  of  enclosure 
(1)  section  6. 

there  are  additional  detailed  recommendations  on  deficiencies 
that  require  correction  or  potential  improvements  that  warrant 
investigation,  these  are  referred  x.o  here. 

When  making  recommendations  as  the  result  of  an  OPEVAL,  use  the 
following  order. 

1.  Recommendation  regarding  ASH. 

2.  Recommendation  regarding  production,  unless  the  first 

recommendation  is  to  grant  full  ASU.  When  full  ASU  is  not  recom¬ 
mended,  specify  C0M0PTEVE0R' s  recommendation  as  to  production 
(e.g.,  do  not  go  into  production,  produce  only  long-lead  items, 
procure  under  a  waiver  of  ASU,  etc.). 

3.  Recommended  correctiv  action. 

4.  Recommendation  regarding  future  0T6E. 
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Copy  to: 

SECNAV  (ASN/R,  E&S) 

(ASN/S&L) 

CNO  (OP-090) 

(OP-90) 

(OP-96) 

(OP-094)  (Fox  C3  Systems) 

(OP-095)  (F OK  ASV  Sf/A-Ce«4) 

(OP-098) 

(OP-98  )  (Cognizant  R DTiE  Division  Dixectox) 

(OP-98T)  (3) 

(OP-13)  (1$  significant  tKai.ni.ng  ok  manpower.  implications) 

(OP-O _ )  (Pxogxam  Sponsox  (DMSO  ok  VCNO)) 

(OP- _ )  (Cognizant  Division  Dixectox (s  j  undcK  SpOnsoxs) 

(OP-  )  (Pxogxam  Cooxdinatox  (when  assigned) ) 

(OP-04)  (Olhexe  OP N  pxocuxement  is  involved) 

R)  CHNAVMAT  (MAT-00) 

(MAT-04) 

(MAT-06) 

(MAT-08D2) 

(MAT-08D1)  (Fok  NAUSEA  pKogKams  only) 

(MAT-08D4)  ( Fok  NAVAIR  pKogKams  only) 

(MAT-08D5)  (Fok  NAUEIEX  pKogKams  only) 

(PM- _ )  (Cognizant  PM,  i{  thexe  is  one) 

(PM- _ )  (Othex  intexestea  PMs,  i£  any) 

(PM-TJ“  (Fox  ASb)  Systems) 

COMNAV _ SYSCOM  ( _ -00)  (Cognizant  Commandex ) 

( _ - _ )  (Cognizant  Deputy  ox  Assistant  Commandex) 

(  )  ( Cognizant  Acquisition  Uanagex) 

(PIP- _ )  ( Cognizant  PMS,  etc.,  i£  thexe  is  one) 

(  ”  )  (Othexs  Ixom  standaxd  d+stxibution  below) 

COMNAVSEASYSCOM  (SEA-90E) 

(SEA-93)  (Fox  suxiace  wax^axe  combatant  ship 

systems) 

(SEA-94)  (Foa  CV,  amphib,  and  auxiliaxy  ship 

systems ) 

COMNAVAIRSYSCOM  (AIR-00) 

(AIR- 06) 

R)  COMN AVELEXS Y SCOM  (ELEX-05E) 

(ELEX-832)  (OPEVAL  KepOKts) 

CINCLANTFLT 

CINCPACFLT 

CINCUSNAVEUR 

7  CLASSIFICATION* 


(Change  4) 


12-16 


S CLASSIFICATION* 


02tcebjr 
3960 (999-OT-I) 


.Copy  to  s  (Cont ' d ) 

COM  LANT  ( Cognizant  type  commanded*) 

COM  PAC  [Nay  be  mote  than  one  In  each  Fleet) 

Participating  Fleet  units,  including  any  Project  Ships  and 
their  ISICs 
DEPCOMOPTEVFORPAC 

COMSURFWARDEVGRU  ( Fok  pKo jecti  Aelated  to  iuA£ace  waA^aAe) 

COMSUBDEVRON  twelve  (F oa  pAojecti  Aelated  to  labmaAlne  waAiaAe) 

CO,  AIRTEVRON  _  (F oa  pAoject i  pAoiecuted  by  a  VXPON) 

OI C,  OPTEVFORDEfr"SUNNYVALE  (F oa  pAojecti  pAoiecuted  by  SUNTEVVET ) 

{Developing  Navy  LaboAatoAy  oa  CenteA ) 
nAV^hiPw^nSysengsta  (Code  6000) 

COMNAVSHIPENGCEN  (Code  6179F) 

OIC,  NAVDET/AFEWC  (ElectAonlc  waA{aAe  OPEVAL  a epoAti) 

CO,  NAVEOTRASUPPCENPAC  ( Foa  pAoject  Involving  acotutlc  itnioAl) 
NAVSEACENLANT  DET  (Code  OOA)  (Foa  oAdnan ce  and  {lAe  contAol 

lyitem  OPEVAL  AepoAti) 

PRESINSURV 
P  RESNAVWARCOL 
CO,  SWOSCOLSCOM 
CNET 
CNA 

COMFLETRAGRU  GTMO 
CO,  FLETRAU  LCREEK 

DTIC  (2)  {Always  lait  on  thh  paAt  o$  the  dhtAlbutlon 

lilt) 


Copy  to  (w/o  enclosure  (1)): 

COMNAVAIRPAC 

COMNAVAIRLANT 

COMNAVSURFLANT 

COMNAVSURFPAC 

COMSUBLANT 

COMSUBPAC 

COMNAVFORCARIB 

COMNAVFORJAPAN 

COMNAVFORMARIANAS 

COMNAVFORKOREA 

COMMIDEASTFOR 


[Eliminate  entAlei  that  aAe 
a epeati  £Aom  ilAit  paAt  oi  Hit) 


{Note:  Thh  Hit  wai  dlAected  by 
the  VCNO) 
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3960 (999-OT-l) 

R)  Copy  to  (w/o  enclosure  (1)):  (Cont'd) 

COMMILSEALIFTCOM 

COMSECONDFLT 

COMTHIRDFLT 

COMSIXTKFLT 

COMSEVENTHFLT 

CG  FMFLANT 

CG  FMFPAC 

COMCARGRU  ONE,  TWO,  THREE,  FOUR,  FIVE,  SIX,  SEVEN,  EIGHT 

COMCRUOESGRU  ONE,  TWO,  THREE,  FIVE,  EIGHT,  TWELVE 

COMSUBGRU  TWO,  FIVE,  SIX,  EIGHT 

COMTRALANT 

COMTRAPAC 

COMSERVGRU  TWO 

COMPHIBGRU  ONE,  TWO 

COHPHIBGRUEASTPAC 

CQMASWWINGSPAC 

COMPATWINGSLANT 

COMSEABASEDASWWINGSLANT 

COMPATWINGSPAC 

COMTACWINGSLANT 

COMF ITAEWWINGP AC 

COMMATVAQWINGPAC 

COMFAIRMED 

COMFAIRWESTPAC 

COMI CEASWGRU 

USCOMSOLANT 

COMINEWARCOM 

COMNAVSURFGRUWESTPAC 

COMNAV  SURFGRUMIDP AC 

CNATRA 

CNTECHTRA 

CO,  AIRTEVRON  ONE,  FOUR,  FIVE 
OIC,  OPTEVFORDET  SUNNYVALE 

Irf  the  Evaluation  Report  heu  no  enclotuxe,  menge  the  two  LUt&, 
with  OTIC  la*t  on  the  combined  lifting. 
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Enclosure  (1) 
CLASSIFICATION* 
REVERSE,  BLANK 

12-19 


CLASSIFICATION* 


Contents  (*) 

Acronyms  and  Abbreviations  ii 

References  ill 

Section  1  —  Description  of  Material 
Section  2  —  Project  Background 
Section  3  —  Scope  of  Evaluation 


301  —  Evaluation  Criteria  3-1 

302  —  Test  Chronology  3-1 

302  —  Limitations  to  Scope  3^1 

Section  4  —  Tests  and  Results 

401  —  Test  E-l,  Low-Altitude  Targets  4-1 


Section  5  —  Operational  Consideration 
Section  6  —  Additional  Recommendations 
Section  7  —  Services  Provided 
Annex  A  —  Instructions  for  Annex  Writing 
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Acronyms  and  Abbreviations  ( * ) 


Operational  Availability 


MTBF 


Mean  Time  Between  Failures 


R  Reliability 

Acronyms  should  he  defined  (spelled  out)  on  the  first  occur¬ 
rence  in  the  text,  and  listed  here.  Two  methods  of  spelling 
out  are  allowed  by  the  Navy  Correspondence  Manual ,  i.e.,  CIC 
(Combat  Information  Center)  or  Combat  Information  Center 
(CIC).  The  former  method  (acronym  first)  is  used  by  COMOPTEVFOR. 

Acronyms  for  naval  activities  included  in  the  Standard 
Naval  Distribution  List  (which  includes  almost  every  activity ) 
need  not  be  spelled  out  or  listed  on  the  acronym  page.  The 
Operational  Test  Director  is  not  precluded  from  spelling  out 
and  listing  such  acronyms,  however,  if  readabi* ity  will  be 
improved  ( e.g .,  acronyms  for  obscure  activities )> 

Acronyms  which  are  defined  (spelled  out)  in  the  letter  need 
not  be  spelled  out  again  in  the  enclosure,  except  on  this 
page. 


ii 
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References  (*) 

(a)  (Secret)  COMOPTEVFOR  XXXXXXZ  Mar  1977,  Quick-look  Report 
on  IOT&E  of  New  Weapons  System  (U) 

(b)  . 

If  references  were  used  in  the  letter a  list  them  here  with 
the  same  letter  designations ,  followed  by  the  first  new 
reference  used  in  the  enclosure ,  etc. 
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Section  1 

Description  of  Material  (*) 

The  purpose  of  this  section  is  to  provide  information 
required  for  completeness ,  but  not  necessary  in  the  body  of 
the  letter.  Examples  of  typical  subparagraph  titles  which 
may  be  required  in  this  section  include: 

101.  Description  of  Equipment.  Provides  details  that  might 
be  useful  at  some  later  date  —  e.g .,  details  regarding  the 
test  installation ,  details  on  how  the  test  system  differed 
from  the  proposed  production  system ,  details  on  the  threat 
simulators  or  targets  that  were  used  in  the  testing.  View 
this  subparagraph  as  a  repository  for  equipment-related 
facts  that  might  be  useful  to  someone  trying  to  determine 
the  source  of  an  equipment  anomaly  later  on  in  a  deployed 
system.  Do  not  repeat  the  equipment  description  in  paragraph 
2  of  the  basic  letter. 

102.  Equipment  Operation.  Records ,  for  possible  future 
use,  the  level  of  skill  and  general  procedures  used  for 
equipment  operation  during  testing.  Specifies  any  differences 
between  operating  procedures  used  during  testing  and  those 
planned  for  deployed  systems. 

103.  Maintenance.  Records  the  same  type  of  information  as 
in  102  above,  only  for  maintenance ,  as  opposed  to  operation. 

104.  Training.  Summarizes  the  formal  and  on-the-job  training 
provided  to  operators  and  maintenance  personnel ,  and  shows 

the  relationship  between  this  training  and  that  planned  for 
fleet  personnel  on  production  equipment. 

105.  Technical  Documentation.  Lists  the  various  operator 
and  maintenance  manuals  and  tactical  guidelines  used  during 
the  testing. 

If  the  basic  letter  says  it  all,  include  this  section  with 
the  notation  "(See  basic  letter)"  directly  under  "Description 
of  Material. " 
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Section  2 

Project  Background  (*) 

Provides  necessary  information  that  supplements  (but  does 
not  repeat)  paragraph  3  of  the  basic  letter.  For  example , 
in  an  OPEVAL  reportt  results  of  TECHEVAL  that  are  pertinent 
to  the  OPEVAL  may  be  listed  here.  If  it  is  not  necessary 
to  supplement  paragraph  3  of  the  basic  lettert  include  this 
section  with  the  notation  "(See  basic  letter)"  directly 
under  " Project  Background. " 
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Section  3 


Scope  of  Evaluation  (*) 


301.  (*)  Evaluation  Criteria.  In  addition  to  the  major 

criteria  listed  in  the  basic  letter,  TEMP  999  provided  the 
following  criteria: 

a.  (*)  Effectiveness 


targets) . 


(1)  (*)  Angular  resolution  —  Y#  (X  square  meter 


(2)  (*)  Range  resolution  —  X  m  (Y  yds) . 


(3)  (*) 


(*)  Suitability 


(1)  (*) 


This  paragraph  expands  on  the  criteria  in  the  baeic  letter 3 
including  the  more  minor  criteria. 

If  no  expansion  is  necessary ,  use  "301.  (*)  Evaluation 

Criteria.  See  basic  letter. " 

302.  (*)  Test  Chronology.  Project  operations  commenced  in 

. . .  on  13  November  1977.  Table  3-1  summarizes 

the  various  tests  including  the  targets  and  ECM  used . . 


This  paragraph  is  an  expansion  of  paragraph  5  of  the  basic 
letter.  Details  such  as  periods  during  which  testing  was 
suspended  (including  full  particulars  regarding  any  Deficiency 
Reports  that  were  issued) ,  dates  of  sorties/ firings ,  etc.t 
should  be  included. 

Test  chronology  is  especially  important  for  projects  that 
involved  extensive  testing  over  long  periods  of  timet  parti¬ 
cularly  when  several  ranges  were  used  or  long  delays  were 
suffered  because  of  deficiencies . 

303.  (*)  Limitations  to  Scope.  In  addition  to  the  major 

limitations  cited  in  the  basic  letter,  testing  was  limited  in 
the  following  ways: 


CLASSIFICATION* 
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a,  (*)  . 

If  no  expansion  on  the  basic  letter  is  required *  use  " 503 . 
(*)  Limitations  to  Scope.  See  basic  letter.” 
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Section  4 


Tests  and  Results  (*) 

If  necessary ,  this  section  may  be  introduced  by  a  " General  Approach " 
paragraph  (new  401).  Typical  content  might  be  a  test  geometry  or 
scenario  that  was  used  throughout  the  testing. 


401. 


a.  (*)  Object.  To  determine  performance  against  low-altitude 
maneuvering  targets  in  the  presence  of  ECM. 


b.  (*)  Procedure 


This  subparagraph  simply  tells  how  the  equipment  was  operated 
and  how  the  data  were  gathered. 


It  may  be  lifted  out  of  the  Test  Plan;  usually  it  is  possible  to 
summarize  the  procedure  of  the  Test  Plan,  however.  For  example , 
the  Test  Plan  may  identify  specific  data  sheets ,  recordings ,  etc, 
to  be  used.  In  the  report  it  is  usually  sufficient  to  say  that 
data  were  recorded  manually  or  automatically ,  etc. 


This  subparagraph  describes  how  the  data  were  analyzed ,  including 
significant  assumptions  and  mathematical  relationships  and 
definitions  of  such  significant  factors  as  success/ failure/no-test, 
material  failures  and  failure  categories ,  and  up  and  down  times. 
This  subparagraph  is  also  based  on  an  equivalent  subparagraph  in 
the  Test  Plan. 


d.  (*)  Results  and  Discussion 

(1)  (*)  In  38  attempted  penetrations,  targets  were  detected 

at  an  average  range  of  ...  (criterion:  ...). 

(2)  (*)  Following  detection,  track  was  established  on  .... 

These  are  the  clear ,  unambiguous  results  of  testing  and  analysis. 
Some  aids  in  preparing  them  follow: 

(1)  Write  them  in  the  past  tense,  and  emphasize  numbers 
rather  than  adj ectives  —  these  two  things  will  help  you  keep 
conclusions  from  creeping  in.  (Conclusions  do  not  belong  in  the 
enc losure.  ) 
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CLASSIFICATION* 


(2)  Don't  let  recommendations  creep  In.  "With,  normal 
lighting  In  the  Apace,  operators  had  difficulty  leading  the  display" 
Is  a  iesu.lt  --It  will  support  an  "additional  recommendation"  later 
In  the  enclosure  that  "action  be  taken  to  Improve  the  readability 
of  the  display."  Don't  foul  up  the  "results"  section  by  putting 
a  recommendation  here  --  Instead,  Identify  the  deficiency  here 
(e.g.,  "A  need  was  Identified  for  a  In  the  past  tense ; 

save  the  recommending  for  later. 


(3)  Summarize  the  data  base  rather  than  presenting  a  mass 
of  raw  data,  but  don't  summarize  so  much  that  you  leave  out  numbers 
completely .  For  example,  consider  a  test  whose  object  Is  to 
determine  the  range  at  which  detection  occurs ;  the  data  base  consists 
of  120  runs  of  a  target  against  the  detection  device.  It  Is  usually 
not  necessary  to  provide  a  tabulation  of  the  detection  range  In 

each  of  the  120  runs.  (If  It  Is  desirable  to  publish  these  run- 
by- run  data,  an  annex  Is  a  better  place  to  put  them.)  I t  Is  usually 
sufficient  to  provide  a  mean  detection  range,  or  a  set  of  means  as 
functions  of  specified  variables  ( e.g .,  with  or  without  active 
jamming),  and  to  specify  the  size  of  the  data  base.  But  don’t  go 
beyond  this  summarization  and  attempt  to  pass  off  a  conclusion  such 
as  "The  system  demonstrated  the  capability  to  detect  at  operationally 
useful  ranges."  Don't  throw  away  M OEs  and  M OSs  In  favor  of 
adjectives.  ^ 

(4)  When  the  data  base  consists  of  questionnaires  filled 
In  by  test  personnel,  remember  that  the  results  that  are  being 
reported  are  results  of  analysis  of  these  questionnaires ,  and 
analysis  Is  a  C0MOPTEVFOR  function  --  not  a  function  to  be  performed 
by  a  reader  of  the  report.  For  this  reason,  do  not  use  statements 
such  as  "Two  of  four  pilots  commented  that  ...."  This  statement 
says  we  didn't  do  our  job  of  analysis  and  follow-up  (interviews , 

etc. )  to  find  out  whether  the  comments  are  valid  or  not.  COMOPTEVFOR 
should  report  that  a  certain  condition  existed,  not  that  a  certain 
percentage  of  people  thought  It  did. 

(5)  When  reporting  results  with  "demonstrated"  values  and 
estimates  at  a  confidence  level,  use  the  following  format  --  It 
avoids  analytic  jargon  (e.g.,  the  lower  one-sided. ..) : 


"The  demonstrated  MT8F  of  system  X  was  227  hours  (criterion: 

>  2 00  hours),  based  on  two  failures  In  454  hours  of  operation. 

At  the  SO  %  confidence  level,  the  actual  MT8F  Is  at  least  105  hours." 

(6)  When  you  compare  a  demonstrated  value  to  an  evaluation 
criterion,  avoid  a  potentially  misleading  statement  such  as  "exceeded 
the  criterion."  It  Is  true  that  the  thoughtful  reader  of  H The 
demonstrated  MTTR  was  2  hours,  which  exceeded  the  0.5 -hour  criterion" 
will  realize  that  repair  took  too  long.  A  hurried  reader,  however,  -.j 
may  draw  a  quick,  wrong  conclusion  --  particularly  If  he's  used  to 
thinking  about  MTBF  and  A-,  which  are  better  when  bigger.  Instead, 
say  "The  demonstrated  MTTR  was  2  hours  (criterion:  < 0.5  hour)." 


(Change  3) 
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(?)  The  total  results  of  thi»  seotion  of  the  enclosure  should 
completely  subetantiatf  the  results  in  the  basic  letter.  But  the 
way  the  results  are  presented  may  differ  between  the  two  parts  of 
the  report  because  of  layout.  Results  in  the  enclosure  are  organized 
by  E-  and  S-tests;  results  in  the  basic  letter  are  organized  by 
objective. 
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Section  5 

Operational  Considerations  (*) 

This  section  is  optional.  If  it  is  not  1 \sedt  assign  section 
number  5  to  the  next  section. 

The  purpose  of  this  section  is  to: 

(1)  Amplify  the  Operational  Considerations  presented 
in  the  basic  letter ,  when  that  ie  necessary .  (Dote  that  use 
of  an  Operational  Considerations  paragraph  in  the  basic 
letter  does  not  make  use  of  this  section  mandatory.) 

(2)  Provide  the  rationale  for  Additional  Recommendations 
in  the  next  section  that  are  based  on  operational  thinking 
rather  than  on  test  results  presented  in  Section  4. 


5-1 


PAGE  5-2, 


CLASSIFICATION* 
REVERSE,  BLANK 


CLASSIFICATION* 


Section  6 


Additional  Recommendations  (*) 

! 

601.  {*)  Additional * Recommendations . 

a.  (*)  Provide  operating  procedures  that: 

(1)  (*)  Contain  pictorial  layouts. 

(2)  (*)  Conform  to  MIL-M-15071G  (NAVY)  in  form  and  format. 

(3)  (*)  .... 

b.  (*)  Make  the  following  changes: 

(1)  (*)  Provide  a  slide-open  cabinet  for  access. 

(2)  (*)  Replace  fasteners  with  easy-to-operate  captive 
fasteners. 


(3)  (*)  .... 

c.  (*)  Investigate  the  feasibility  of: 

(1)  (*)  Adding  a  tape  reader  at  Station  No.  2. 

(2)  (*)  Using  a  standard  Navy  lubricant  instead  of  the 
proprietary  lubricant  that  was  required  by  the  test  system. 

This  is  alao  an  optional  eeotion .  (Note  that  if  there  is  no  Opera* 
tional  Conaideratione  section^  thie  aeotion  is  numbered  Section  5. ) 
The  purpoae  of  thia  8ection  i8  supplement  paragraph  9  of  the  basic 
letter  with  recommendatione  that  are  not  individually  of  interest 
at  the  decision~making  level. 


I  ' 
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Section  7 

Services  Provided  (*) 

This  section  provides  a  record  for  future  use  in  estimating  costs 
of  OTSE.  Include  here  in  tabular  or  other  convenient  form  the 
services  provided  during  the  phase  of  testing  being  reported. 
Services  include  such  things  as  dedicated  and  not-to-interfere  ship 
8upportt  test  aircraft i  targets ,  and  operating  personnel .  One  page 
should  usually  suffice  to  record  this  information. 
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Annex  A 

Ins  try  cl  Lon  s  f  or  Annex  Wri-tiny 

Annexes  present  material  pertinent  to  the  evaluation,  but 
not  appropriate  for  inclusion  in  enclosure  (1)  because  of 
length  or  detail.  Such  material  would  be  individual  firing 
summaries,  as  opposed  to  the  integrated  and  summarized  data 
presented  in  Section  4.  Pertinent  reports  from  other 
commands,  etc.,  may  be  included. 

Annexes  must  be  referred  to  in  the  text  of  the  enclosure, 
and  listed  on  the  contents  page. 
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The  Quick- look  Report 


a.  Usually  a  Quick- look  Report  and  the  letter  of  the 
Report  which  cancels  and  supersedes  it  will  be  essentially 
identical.  When  you  have  written  the  first,  the  second  is 
almost  done  —  there  is  usually  no  need  to  rephrase  "Equipment 
Description"  etc.  —  just  add  the  "the's"  and  whatnot  to 
make  prose  out  of  the  message.  If  substantive  differences 
will  exist  between  the  two  —  e.g.,  if  post-Quick- look 
failure  analysis  causes  reliability  figures  to  change  — 
these  differences  must  be  identified  and  explained  in  the 
letter.  For  example  "The  demonstrated  MTBF  of  system  X  was 
227  hours,  based  on...  This  MTBF  is  less  than  was  reported 
in  reference  (a)  because  one  failure  was  detected  during 
data  analysis  after  reference  (a)  had  been  issued.  At  the 
80%  confidence  level,  the  actual  MTBF  is  at  least  86  hours." 


b.  A  sample  Quick- look  Report  is  provided  on  the  following 
pages . 
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R15Q422  JUL  66 
FM  COMOPTEVFOR  NORFOLK  VA 
TO  CNO  WASHINGTON  DC 

INFO  CHNAVMAT  WASHINGTON  DC  COMNAVSURFLANT  NORFOLK  VA 
CINCLANTFLT  NORFOLK  VA  COMSECONDFLT  NORFOLK  VA 
USS  HEWES  NUC  CHINA  LAKE  CA 

UNCLASSIFIED//N03U0// 

GUICK-LOOK  REPORT  OF  OPEVAL  OF  SHIPBOARD  CERBERUS  MISSILE 
SYSTEM 

A.  NWC  REPORT  TR-XXX  OF  26  SEP  fit 

B-  COMOPTEVFOR  LTR  SER  YYY  OF  21  JUN  fit 

C.  TEST  AND  EVALUATION  MASTER  PLAN  NO.  XXX  OF  17  MAR  fib 

1.  SUMMARY.  THIS  IS  GUICK-LOOK  REPORT  ON  OPEVAL  -COPERA- 
TIONAL  EVALUATION!  OF  SHIPBOARD  VERSION  OF  CERBERUS  MISSILE 
SYSTEM-.  PERFORMED  UNDER  CNO  PROJECT  XXX-OT-IIA.  BASED  ON 
COMPLETE  PERFORMANCE  ANALYSIS  AND  PRELIMINARY-.  INCOMPLETE 
SUITABILITY  ANALYSIS-.  SHIPBOARD  CERBERUS  DETERMINED  TO  BE 
OPERATIONALLY  EFFECTIVE  AND  POTENTIALLY  OPERATIONALLY  SUIT¬ 
ABLE.  PROVISIONAL  APPROVAL  FOR  SERVICE  USE  AND  LIMITED 
PRODUCTION  RECOMMENDED.  END  SUMMARY- 

2.  SYSTEM  DESCRIPTION 

A.  CERBERUS  MISSILE  SYSTEM  IS  ANTI-SURFACE-SHIP  WEAPON 
SYSTEM  CONSISTING  OF  15Q-NMI-RANGE  AIR-BREATHING  RF/IR 
{RADIO  FREGUENCY/INFRARED1  HOMING  MISSILE-.  WEAPONS  CONTROL 
SYSTEM-,  AND  LAUNCHER.  CERBERUS  HAS  THREE  VARIANTS:  SHIP--. 
AIR--.  AND  SUBMARINE-LAUNCHED.  THIS  TESTING  WAS  OF  SHIP- 
LAUNCHED  CERBERUS  INSTALLED  USS  HEWES  CFF  10761. 

B-  CERBERUS  IS  TARGETED  FROM  EXTERNAL  SOURCES  {NTDS 
{NAVY  TACTICAL  DATA  SYSTEM!  IN  HEWES!-  ALIGNMENT  DATA  FOR 
MISSILE'S  INERTIAL  MIDCOURSE  GUIDANCE  SYSTEM  ARE  PROVIDED  BY 
SINS  {SHIP'S  INERTIAL  NAVIGATION  SYSTEM!.  NTDS  AND  SINS 
DATA  ARE  PROCESSED  BY  WEAPONS  CONTROL  SYSTEM'S  LCC  {LAUNCH 
CONTROL  CONSOLE!-.  WHICH  CALCULATES  FLIGHT  PATH  TO  DESIGNATED 
TARGET  THAT  MINIMIZES  CHANCES  OF  HITTING  FORBIDDEN  TARGETS 
EN  ROUTE-.  AND  PROGRAMS  MISSILE  ACCORDINGLY-  LAUNCH  IS 
OPERATOR-INITIATED  AT  LCC.  FOR  THIS  TESTING-.  TWO-TUBE 
LAUNCHER  WAS  MOUNTED  FORWARD  IN  HEWES. 


3 .  BACKGROUND 


A.  CERBERUS  MISSILE  SYSTEM  UAS  DEVELOPED  IN  RESPONSE  TO 
OPERATIONAL  REQUIREMENT  0R-007X. 

B.  OTgE  {OPERATIONAL  TEST  AND  EVALUATION!  OF  CERBERUS 
BEGAN  IN  l*1fl4  {OT-I!.  LCC  TARGETING  AND  INITIALIZATION 
CAPABILITIES  WERE  VERIFIED-.  AS  WERE  MILESTONE  II  THRESHOLDS 
FOR  ON-BOARD  SYSTEM  RELIABILITY,  MAINTAINABILITY,  AND  AVAIL¬ 
ABILITY. 

C.  MISSILE  SURVIVABILITY  DURING  FLIGHT  UAS  ASSESSED  AS 
HIGH  BY  NUC  CHINA  LAKE  IN  REF  A. 

D-  CAPABILITY  OF  EXTERNAL  SOURCES  TO  PROVIDE  TARGETING 
DATA  UAS  VERIFIED  IN  RELATED  OT&E  UNDER  CNO  PROJECT  YYY-OT- 
IIB  -CSEE  REF  B>. 

E-  CERBERUS  OPEVAL  BEING  CONDUCTED  IN  THREE  CONCURRENT 
PHASES:  OT-IIA  FOR  SHIPBOARD  VERSION,  OT-IIB  FOR  AIR- 
LAUNCH,  OT-IIC  FOR  SUB-LAUNCH. 

4.  SCOPE 

A.  OBJECTIVES.  OBJECTIVES  OF  PROJECT  XXX-OT-IIA  UERE 
TO  DETERMINE: 

{!>  PROBABILITY  THAT  SYSTEM  UILL  BE  AVAILABLE, 
LAUNCH,  AND  DETONATE  ON  DESIRED  TARGET- 

{S!  CAPABILITY  OF  SYSTEM  TO  READY  CERBERUS  FOR 
LAUNCH  AT  DESIRED  TIME. 

{3}  CAPABILITY  OF  MISSILE  TO  HIT  TARGET- 

{4}  PROBABILITY  OF  NOT  HITTING  FORBIDDEN  TARGET- 

{5!  PROBABILITY  OF  UARHEAD  DETONATION  {GIVEN  HIT}. 

{b!  SURVIVABILITY/VULNERABILITY  OF  SYSTEM. 

{?}  ADEQUACY  {VALIDATION!  OF  CERBERUS  FLIGHT  SIMU¬ 
LATION. 

{*}  SYSTEM  RELIABILITY,  MAINTAINABILITY,  AND  AVAIL¬ 
ABILITY. 

{*1!  LOGISTICS  SUPPORTABILITY  IN,  AND  COMPATIBILITY 
UITH,  SHIPBOARD  ENVIRONMENT. 


{10}  INTEROPERABILITY  WITH  NTDS  AND  SINS. 

{11}  ADEQUACY  OF  PLANNED  TRAINING- 

{12}  TRANSPORTABILITY  OF  CERBERUS  MISSILE,  AND  MIS¬ 
SILE  SAFETY  IN  SHIPBOARD  STORAGE- 

{13}  ADEQUACY  OF  HUMAN  ENGINEERING  DESIGN- 

B-  EVALUATION  CRITERIA-  FOLLOWING  CRITERIA  PROVIDED 
REF  C: 

{1}  OVERALL  MOMS  {MEASURE  OF  MISSION  SUCCESS}: 
PROBABILITY  OF  PROPER  TARGETING,  LAUNCH,  HITTING  TARGET, 
DETONATION,  DAMAGE  ASSESSMENT  —  Q.fcH. 

{2}  FOLLOWING  TARGET  DESIGNATION,  MISSILE  READIED 
FOR  LAUNCH  IN  30  SECONDS  OR  LESS  WITH  PROBABILITY  OF  0-15- 

{3}  GIVEN  LAUNCH,  PROBABILITY  OF  HITTING  TARGET  — 

0-74- 

{4}  PROBABILITY  OF  NOT  HITTING  FORBIDDEN  TARGET  — 

0-  TS- 

{5}  GIVEN  HIT,  PROBABILITY  OF  WARHEAD  DETONATION  — 

0.15. 

{b}  MISSILE  SURVIVAL  RATE  IN  LAUNCHER/MAGAZINE 
STORAGE  —  O.flO  FOR  a  MONTHS- 

{?}  ON-BOARD  SYSTEM  fi-HOUR  MISSION  RELIABILITY  — 

0-^4- 

{3}  ON-BOARD  SYSTEM  MTTR  {MEAN  TIME  TO  REPAIR}  — 

1-5  HOURS- 

{1}  MISSILE  AVAILABILITY  AT  LAUNCH  COMMAND  —  O-'H- 
{10}  ON-BOARD  SYSTEM  OPERATIONAL  AVAILABILITY  -- 

O-'IS- 

C-  LIMITATIONS  TO  SCOPE 

{1}  ACTUAL  AND  FORBIDDEN  TARGETS  SIMULATED  BY  HULKS/ 
BARGES  EQUIPPED  WITH  RF  AND  IR  SOURCES-  SIMULATIONS  NOT 
FULLY  REPRESENTATIVE  OF  ACTUAL  SHIPS- 

{2}  PROBABILITIES  OF  HITTING  TARGET  AND  NOT  HITTING 
FORBIDDEN  TARGETS  ESTIMATED  PRIMARILY  ON  BASIS  OF  NON-FIRING 
EXERCISES  AND  COMPUTER  SIMULATION. 
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•C3J  PROBABILITY  OF  PROPER  DAMAGE  ASSESSMENT  NOT 
TESTED.  ASSUMED  TO  BE  1.00  IN  MEASURE  OF  MISSION  SUCCESS 
CALCULATIONS. 

{4!  EFFECTS  OF  ENVIRONMENTAL  EXTREMES  NOT  TESTED- 
S.  PROJECT  OPERATIONS 

A.  OT-IIA  OPERATIONS  WERE  CONDUCTED  ABOARD  HEWES  IN 
VIRGINIA  CAPES  AND  ROOSEVELT  ROADS  OPERATING  AREAS  FROM  7 
JUN  TO  5  JUL  aa. 

•Cl>  SEVENTY-EIGHT  NON-FIRING  EXERCISES  WERE  CONDUC¬ 
TED.  TARGETS  IN  NTDS  DATA  BASE  WERE  DESIGNATED-.  FORBIDDEN 
TARGETS  WERE  ASSIGNED-.  AND  MISSILES  WERE  PREPARED  FOR  LAUNCH. 
LCC  RECORDINGS  OF  MISSILE  PARAMETERS-,  NTDS  GROUND  TRUTH-.  AND 
RECORDED  SINS  DATA  WERE  EXERCISED  ON  NWC’S  CERBERUS  FLIGHT 
SIMULATION  PROGRAM  TO  RECONSTRUCT  TIMING  OF  MISSILE  PREPARA¬ 
TION-.  AND  TO  PREDICT  PROBABLE  OUTCOME  OF  SIMULATED  LAUNCHES. 

{2!  THREE  LIVE  FIRINGS  -CNON-UARHEAD>  WERE  CONDUCTED. 

{3!  LCC  MAINTAINED  IN  ALERT  STATUS  -COR  HIGHER!  FOR 
LEO  HOURS  DURING  PROJECT  OPERATIONS. 

B.  DATA  FROM  E2  MISSILE  FIRINGS  IN  DT-II,  OT-IIB-.  AND 
OT-IIC  USED  TO  SUPPLEMENT  OT-IIA  DATA  BASE  FOR  ANALYSIS. 

C.  CERBERUS  FLIGHT  SIMULATION  PROGRAM  WAS  VERIFIED 
USING  RESULTS  OF  LIVE  SURFACE  LAUNCHES  IN  DT/OT-II. 

PROGRAM  WAS  THEN  EXERCISED  APPROXIMATELY  EDO  TIMES  TO  DERIVE 
MOMS. 

b.  RESULTS 

A.  MISSION  SUCCESS.  BASED  ON  COMBINATION  AT-SEA  FIRINGS 
AND  CERBERUS  FLIGHT  SIMULATION-,  MOMS  WAS  0-7E  FOR  BARRIER 
PATROL  AND  TRANSIT  OPERATION  SCENARIOS.  MOMS  FOR  SELECTIVE 
ATTACK  SCENARIOS  VARIED  FROM  O.bS  TO  0-83  -[CRITERION  0.b2>. 

B.  TIME  TO  LAUNCH 

{1!  ON  BASIS  OF  78  NON-FIRING  EXERCISES-,  DEMONSTRA¬ 
TED  PROBABILITY  OF  LAUNCHING  IN  30  SECONDS  OR  LESS  WAS  0.17 
t?b  OF  ?a>  {CRITERION  0-15!.  FAILURES  TO  MEET  CRITERION  <32 
AND  34  SECONDS!  APPEARED  ASSOCIATED  WITH  HEWES  MANEUVERING 
AND  ROUGH  SEAS^  CAUSING  RAPIDLY  FLUCTUATING  SINS  AND  MISSILE 
MIDCOURSE  GUIDANCE  SIGNALS. 

{2!  DATA  ANALYSIS  INDICATED  TIME  TO  ALIGN  CERBERUS 
MIDCOURSE  GUIDANCE  SYSTEM  MAY  BE  EXCESSIVE  <UP  TO  30  MINUTES! 


IN  SOME  SEA  CONDITIONS.  PROBLEM  MAY  BE  ASSOCIATED  WITH 
FREQUENCY  OF  WAVE  ACTION,  DETAILS  WILL  BE  PROVIDED  IN  FORMAL 
EVALUATION  REPORT. 

C-  CAPABILITY  TO  HIT  TARGET 

{1!  ON  BASIS  OF  NON-FIRING  EXERCISES,  ESTIMATED 
PROBABILITY  OF  HITTING  TARGET  {ASSUMING  NO  FAILURES!  WAS 
O.IS  {74  OF  7fi}.  FOUR  MISSES  APPEARED  ASSOCIATED  WITH  BOW- 
ON  TARGET  ASPECT  IN  ROUGH  SEAS,  WHICH  DEFEATED  IR  HOMING 
LOGIC- 

{2!  ON  BASIS  OF  fl  SURFACE  LAUNCHES  {INCLUDING  S  FROM 
DT-III,  DEMONSTRATED  PROBABILITY  OF  HITTING  TARGET  WAS  1.0 
{CRITERION  0*74}. 

D.  CAPABILITY  TO  NOT  HIT  FORBIDDEN  TARGETS 

{1}  ON  BASIS  OF  NON-FIRING  EXERCISES,  ESTIMATED 
PROBABILITY  OF  NOT  HITTING  FORBIDDEN  TARGET  WAS  O-Tl  {77  OF 
7fi! •  OF  4  SIMULATED  LAUNCHES  THAT  FAILED  TO  HIT  DESIGNATED 
TARGET,  3  IMPACTED  WATER  CLOSE  ABOARD  TARGET.  FOURTH  OVER¬ 
FLEW  TARGET  AND  ERRONEOUSLY  ACQUIRED  FORBIDDEN  TARGET  WITH 
IR  SEEKER.  RF  SIGNAL  CORRELATION  DID  NOT  PRECLUDE  HOMING, 
BECAUSE  IR  HOMING  PREEMPTS  RF  HOMING. 

{5>  ON  BASIS  OF  7  DT/OT-II  SURFACE  LAUNCHES  WITH 
FORBIDDEN  TARGETS  IN  TARGET  AREA,  DEMONSTRATED  PROBABILITY 
OF  NOT  HITTING  A  FORBIDDEN  TARGET  WAS  1-0  {CRITERION  0.*15}. 

E.  WARHEAD  DETONATION 

{1}  IN  22  NON-WARHEAD  FIRINGS  DURING  DT/OT-II, 

THERE  WERE  21  CASES  WHERE  PROPER  FUZING  WAS  VERIFIED. 
TELEMETRY  FAILURE  PRECLUDED  VERIFICATION  DURING  ONE  FIRING. 
BASED  ON  FUZE  ACTUATION,  ESTIMATED  PROBABILITY  OF  WARHEAD 
DETONATION  WAS  1.0  {21  OF  21>. 

{2>  THREE  WARSHOTS  HAVE  BEEN  FIRED*,  TWO  DURING  DT- 
IIC  {SUBMARINE  LAUNCH!,  ONE  DURING  OT-IIB  {AIR  LAUNCH!. 

ALL  DETONATED,  FOR  A  DEMONSTRATED  PROBABILITY  OF  DETONATION 
{GIVEN  HIT!  OF  1.0  {CRITERION  O.'JS!. 

F.  SURVIVABILITY/VULNERABILITY.  NO  MAJOR  DEFICIENCIES 
NOTED. 

G.  CERBERUS  FLIGHT  SIMULATION-  HIGH  CORRELATION  OF 
HIT/MISS  WITH  ACTUAL  MISSILE  FIRINGS. 


H.  RELIABILITY 


•C1J  DURING  DT/OT-II ■.  ID  CERBERUS  MISSILES  -CS  FOR 
SURFACE  LAUNCH-.  S  FOR  AIR  LAUNCH!  WERE  SUBJECTED  TO  TOTAL  OF 
APPROXIMATELY  1D20  DAYS  OF  SHIPBOARD  STORAGE  IN  MAGAZINE  OR 
LAUNCHER.  ONE  MISSILE  FAILED  {MIDCOURSE  GUIDANCE}-,  FOR 
DEMONSTRATED  MTBF  {MEAN  TIME  BETWEEN  FAILURES}  OF  1020  DAYS- 
GIVEN  THIS  MTBF •,  DEMONSTRATED  PROBABILITY  OF  SURVIVING  fi 
MONTHS  OF  STORAGE  WAS  0.7,1  {CRITERION  O.flO}. 

{2}  DURING  OT-IIAi  LCC  WAS  OPERATED  {ALERT  AND 
ABOVE!  FOR  L20  HOURS.  TWO  FAILURES  OCCURRED  {BOTH  IN  COM¬ 
PUTER/PROCESSOR}-,  FOR  DEMONSTRATED  LCC  MTBF  OF  310  HOURS. 
LAUNCHER  SUSTAINED  NO  FAILURES.  BASED  ON  THESE  DATA-,  DEMON¬ 
STRATED  fi-HOUR  MISSION  RELIABILITY  OF  ON-BOARD  SYSTEM  WAS 
O.6)?  {CRITERION  O.'JflM}. 

I.  MAINTAINABILITY.  DEMONSTRATED  MTTR  OF  ON-BOARD 
SYSTEM  WAS  1-2  HOURS  {CRITERION  1.5  HOURS}.  MTTR  VALUE 
DERIVED  FROM  2  ACTUAL  REPAIR  ACTIONS-,  AND  12  ACTIONS  RESULT¬ 
ING  FROM  INSERTION  OF  PRE-FAULTED  MODULES. 

J.  AVAILABILITY 

{1}  IN  DT/OT-II-,  13  MISSILES  WERE  SUBJECTED  TO 
LAUNCH  COMMAND  IN  SURFACE  LAUNCHER.  ALL  WERE  LAUNCHED 
SUCCESSFULLY-,  FOR  DEMONSTRATED  MISSILE  AVAILABILITY  AT 
LAUNCH  OF  1.0  {CRITERION  0. *!*?}. 

{2}  DURING  OT-IIA-,  ON-BOARD  SYSTEM  WAS  UP  FOR  bfl3 
HOURS-,  DOWN  FOR  2-5  HOURS-,  FOR  DEMONSTRATED  OPERATIONAL 
AVAILABILITY  APPROACHING  1-0  {CRITERION  0.^5}. 

K.  LOGISTICS  SUPPORT  ABILITY  -,  COMPATIBILITY-,  AND  INTER¬ 
OPERABILITY.  NO  MAJOR  DEFICIENCIES  WERE  NOTED  IN  THESE 
AREAS.  MINOR  DEFICIENCIES  WILL  BE  DISCUSSED  IN  FORMAL 
EVALUATION  REPORT. 

L.  TRAINING.  DRAFT  NAVY  TRAINING  PLAN  APPEARED  ADE¬ 
QUATE  FOR  OPERATORS  AND  MAINTENANCE  PERSONNEL- 

M.  TRANSPORTABILITY.  CERBERUS  MISSILES  WERE  TRANSPOR¬ 
TED  AND  DELIVERED  BY  UNDERWAY  AND  VERTICAL  REPLENISHMENT 
DURING  DT-IID •  NO  PROBLEMS  WERE  NOTED. 

N.  SAFETY.  NO  SAFETY  PROBLEMS  ASSOCIATED  WITH  HAND¬ 
LING-,  TRANSPORTING-,  OR  STORING  CERBERUS  MISSILES  HAVE  BEEN 
NOTED- 

O.  HUMAN  ENGINEERING 
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m  ON  LCC -i  SOME  FUNCTIONS  SHIFTED  FROM  ONE  BUTTON 
POSITION  TO  ANOTHER  AS  MODE  OF  OPERATION  CHANGED •  THIS  IS 
CONDUCIVE  TO  INPUT  ERROR  UNDER  STRESS  CONDITIONS. 

{21  MODE/STATUS  INDICATORS  ON  LCC  WERE  TOO  BRIGHT- 
THEY  OBSCURED  ALPHANUMERICS  AND  CONTRIBUTED  TO  EYE 
FATIGUE- 

7.  OPERATIONAL  CONSIDERATIONS 

A.  MISSILE  RELIABILITY.  ONLY  MISSILE  FAILURE  WAS 
DETECTED  DURING  CHECKOUT  ABOARD  SCHEDULED  LAUNCH  AIRCRAFT. 

MISSILE  HAD  PASSED  IDENTICAL  CHECK  THE  PREVIOUS  DAY-.  AFTER 
APPROXIMATELY  120  DAYS  OF  SHIPBOARD  MAGAZINE  STORAGE  AND  lfl 
EARLIER  SUCCESSFUL  ON-BOARD- AIRCRAFT  CHECKS.  BECAUSE  THESE 
CHECKS  MAY  CONTRIBUTE  TO  MISSILE  FAILURES-.  AND  BECAUSE  RATE 
OF  THESE  CHECKS  EXCEEDED  ANTICIPATED  OPERATIONAL  CHECKOUT 
RATE  OF  ONE  PER  30  DAYS-.  FAILURE  TO  MEET  fl-MONTH  MISSILE 
SURVIVAL  RATE  -CO-TT  VERSUS  D-flO  CRITERION}  NOT  CONSIDERED 
SIGNIFICANT. 

B-  ON-BOARD  SYSTEM  RELIABILITY.  ACCORDING  TO  CONTRAC¬ 
TOR-,  BOTH  LCC  COMPUTER/PROCESSOR  FAILURES  MAY  HAVE  HAD 
IDENTICAL  CAUSE  {OVERLOADED  TRANSISTOR  IN  TIMING  CIRCUIT}. 

BOTH  FAILURES  HAVE  BEEN  REPRODUCED-,  AND  FIX  HAS  BEEN  INSTAL¬ 
LED  AND  VERIFIED  IN  CONTRACTOR'S  LABORATORY. 

A.  CONCLUSIONS.  SHIPBOARD  VERSION  OF  CERBERUS  MISSILE 
SYSTEM: 

A.  IS  OPERATIONALLY  EFFECTIVE-,  BASED  ON  DEMONSTRATED 
CAPABILITY  TO  HIT  DESIGNATED  TARGETS  WHILE  AVOIDING  FORBID¬ 
DEN  TARGETS. 

b.  HAS  POTENTIAL  TO  BE  OPERATIONALLY  SUITABLE-,  PROVIDED 
LCC  RELIABILITY  AND  HUMAN  ENGINEERING  DEFICIENCIES  ARE  ELIMINATED. 

1 •  RECOMMENDATIONS 

A.  PROVISIONALLY  APPROVE  SHIPBOARD  CERBERUS  MISSILE 
SYSTEM  FOR  SERVICE  USE  AFTER: 

{1}  ENSURING  THAT  CAUSEIS}  OF  LCC  COMPUTER/PROCESSOR 
FAILURES  HAS  BEEN  ELIMINATED- 

{2}  ELIMINATING  FUNCTION  SHIFT  ON  LCC  BUTTONS  DURING 
MODE  CHANGES. 
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<3>  REDUCING  INTENSITY  OF  LCC  NODE/STATUS  INDICATORS 
{SUGGEST  A  DIMMER  SWITCH}. 

8.  PROCEED  WITH  PLANNED  FY&fi  BUY  OF  SHIPBOARD  CERBERUS 
MISSILE  SYSTEMS. 

C.  CONDUCT  OT-III  AS  PREREQUISITE  TO  APPROVAL  FOR  SER¬ 
VICE  USE-*  TO  VERIFY  SYSTEM  OPERATIONAL  SUITABILITY. 

D-  INVESTIGATE  POSSIBILITY  THAT  MISSILE  ALIGNMENT  TIMES 
MAY  BE  EXCESSIVE  IN  SOME  SITUATIONS. 


1208.  Using  Contractor  Support  in  Writing  Reports 


a.  If  you  use  contractor  support  in  writing  reports, 
make  sure  the  contractor  has  an  up-to-date  copy  of  this 
Guide  —  so  he  knows  how  the  report  should  be  written. 

b.  Emphasize  to  him  that  on  this  job  there's  no 
payment  by  the  pound. 

c.  Review  the  product  as  it  is  being  produced  to  make 
sure  it's  on  track. 


d.  Tell  him  not  to  hesitate  to  call/visit  the  Staff 
Technical  Editor  if  he  has  a  problem. 


1209.  Preparing  and  Staffing  Reports  on  DEPCOMOPTEVFORPAC 
Projects.  In  order  to  minimize  the  time  required  to  process 
reports  on  projects  assigned  to  DEPCOMOPTEVFORPAC,  the 
following  procedures  are  used: 

a.  During  project  operations,  and  upon  their  completion, 
the  Headquarters  OTC  visits  Deputy  to  become  aware  of  pre¬ 
liminary  results.  As  appropriate,  areas  of  concern  in  the 
Evaluation  Report  are  discussed. 


t  When  Deputy  has  a  working  draft  of  the  report,  as  many 
Deputy  staff  members  as  are  required  come  to  Headquarters,  and 
a  common  working  draft  is  prepared.  This  working  draft  is 
reviewed  in  parallel  in  the  Headquarters,  through  the  02  level. 


c.  The  common  working  draft  will  be  returned  to  Deputy. 
Any  remaining  issues  will  be  resolved  by  the  Deputy 
Commander  with  the  Force  Commander  before  he  issues  the 
recommended  draft  for  signature. 


1210.  Checklists.  The  attached  sheets  are  designed 
to  help  the  OTD  avoid  some  of  the  most  frequent  report 
errors . 
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QUICK-LOOK  CHECKLIST 


1.  Summary  paragraph  actually  summarizes,  including 

conclusions  and  recommendations.  (  ) 

2.  Equipment  description  says  what  the  thing  is 

supposed  to  do.  (  ) 

3.  Background  says  why  someone  thinks  the  Navy  needs 

the  thing.  (  ) 

4.  Objectives  are  listed  in  the  proper  order  (opera¬ 

tional  effectiveness  ones  first,  then  reliability, 
maintainability,  etc.).  (  ) 

5.  Limitations  are  limitations  to  the  evaluation, 

not  how  hard  it  was.  (  ) 

6.  Evaluation  criteria  are  in  the  same  order  as 

objectives.  (  ) 

7.  Testing  gives  an  idea  of  the  way  testing  was 
conducted  (scenarios),  and  how  much  was  done  (number 

of  bombs ,  etc . ) .  (  ) 

8.  Results  address  objectives  (and  associated  cri¬ 
teria)  in  the  same  order  as  they  are  presented.  (  ) 

9.  All  objectives  (and  criteria)  are  addressed, 

except  as  noted  in  limitations  to  scope.  (  ) 

10.  If  operational  considerations  are  included, 

they  assist  in  going  from  results  to  conclusions  and 
recommendations .  (  ) 

11.  Conclusions  address  operational  effectiveness 

first,  then  operational  suitability.  (  ) 

12.  Conclusions  derive  from  results  —  no  hardware 

mentioned  for  the  first  time,  no  discrepancies  iden¬ 
tified  for  the  first  time,  etc.  (  ) 

13.  Recommendations  address  the  milestone  (full- 

scale  development,  production,  etc.).  (  ) 

14.  Recommendations  derive  from  conclusions.  (  ) 


LETTER  REPORT  CHECKLIST 


1.  Purpose  is  the  standardized  paragraph  of  the 

sample  report  in  the  OTD  Guide.  (  ) 

2.  Equipment  description  emphasizes  the  function  of 
the  equipment,  and  how  equipment  tested  differed  from 

the  planned  configuration.  (  ) 

3.  Background  summarizes  key  elements  of  the  develop¬ 
ment,  with  emphasis  on  results  of  prior  T&E.  (  ) 

4.  Objectives  are  in  the  proper  order  (operational 
effectiveness,  first,  followed  by  operational  suita¬ 
bility  objectives  (reliability,  maintainability, 

etc. )  (  ) 

5.  Evaluation  criteria  quantify  (and  do  not  simply 

repeat)  objectives.  (  ) 

6.  Limitations  to  scope  clearly  describe  actual  limi¬ 
tations  to  the  evaluation .  (  ) 

7.  Project  operations  provides  insight  into  the 

operational  realism  and  amount  of  testing.  (  ) 

8.  Results  are  keyed  to  objectives,  in  the  same 
order  as  objectives,  and  address  all  objectives  and 
evaluation  criteria  (unless  exempted  by  limitations 

to  scope ) .  (  ) 

9.  Results  do  not  conclude.  (  ) 

10.  Operational  considerations  (if  included)  discuss 

operational,  aspects  which  influence  interpretation  of 
results,  and  conclusions.  (  ) 

11.  Conclusions  address  operational  effectiveness 

first,  then  operational  suitability.  (  ) 

12.  Conclusions  don't  introduce  new  thoughts  —  no 

hardware  is  mentioned  for  the  first  time,  no  discrep¬ 
ancies  are  identified  for  the  first  time,  etc.  (  ) 

13.  Recommendations  address  the  milestone  (full- 

scale  development,  approval  for  sevice  use  and  pro¬ 
duction,  etc.).  (  ) 


14.  There  are  no  new  thoughts  in  recommendation. 

15.  If  there  was  a  quick-look,  any  differences 
between  this  and  the  quick-look  are  identified  and 
explained. 
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ENCLOSURE  CHECKLIST 

X.  Enclosure  amplifies  —  does  not  repeat  —  letter. 

2.  Enclosure  has  no  conclusions  buried  in  it  — 
disguised  as  results,  operational  considerations, 
etc. 

3.  Enclosure  has  only  additional  recommendations  (in 
self-contained  section). 

4.  Enclosure  is  fully  consistent  with  letter. 
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Section  13 
Tactics  Guides 

1301.  Introduction 

a.  A  major  function  of  OT&E  is  the  assessment  of  tac¬ 
tics  for  employment  of  new  weapon  systems.  Weapons  system 
tactics  are  published  in  OTGs  (OPTEVFOR  Tactics  Guides)  by: 

(1)  Commanding  Officers  of  VX  Squadrons,  on  subjects 
under  their  cognizance. 

(2)  COMOPTEVFOR,  on  all  other  appropriate  subjects. 

b.  The  TEMP  identifies  the  tactics  development  aspects 
of  a  T&E  project  as  follows: 

(1)  Tactics  development  is  specified  as  an  objective 
of  each  appropriate  phase  of  future  OT&E  in  Part  IV  of  the 
TEMP. 

(2)  Anticipated  OTG  publication  dates  are  shown  in 
the  OT&E  Reports  line  of  Part  II  of  the  TEMP. 

1302.  Types  of  OPTEVFOR  Tactics  Guides 

a.  OPTEVFOR  Preliminary  Tactics  Guides  provide  early 
information  on  tlctical  employment  of  new  weapons  systems 
entering  or  in  early  stages  of  full-scale  development.  They 
are  prepared  at  the  conclusion  of  OT-I  or  early  sub-phases 
of  OT-I I. 

b.  OPTEVFOR  Tactics  Guides  provide  baseline  tactics  for 
employment  or  new  weapons  systems.  They  are  prepared  at  the 
conclusion  of  OPEVAL. 

c.  OPTEVFOR  Follow-on  Tactics  Guides  provide  refined 
tactical  information  on  new  weapons  systems  actually  in 
production.  They  are  prepared  at  the  conclusion  of  OT-III. 

1303.  The  Elements  of  an  OPTEVFOR  Tactics  Guide.  OPTEVFOR 
Tactics  Guides  are  designed  to  provide  tSST  fleet  user  with 
the  following  types  of  information: 

a.  Operational  capabilities  of  the  equipment.  What 
will  it  do  for  the  user  —  in  operational  terms.  OT&E  may 
tell  what  an  equipment  does  against  some  spec  that  means 
something  to  an  engineer.  OT&E  tells  what  it  will  do  for 
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that  an  ESM  (electronic  support  measure)  system  has  a  re¬ 
ceiver  sensitivity  of  X  dB  —  OT&E  says  what  it  will  do 
against  specific  Soviet  emitters  when  its  antenna  is  mounted 
so  high.  Operational  capabilities  include  operating  proce¬ 
dures  that  tell  you  how  to  get  the  most  out  of  the  equip- 

ment  —  e.g.f  if  you  want  to  listen  at  _  MHz,  secure  the 

_ _ .  Operating  procedures  do  not  tell  how  to  turn  the 

equipment  on  and  how  to  tune  it  —  they  do  not  substitute 
for  operator  manuals. 

b.  Tactical  concepts  —  not  pat  solutions  to  big  prob¬ 
lems,  but  rather  starting  points  for  the  user's  thinking. 
Building  blocks,  or  small  pieces  of  the  problem.  What 
sonobuoy  pattern  worked  best  under  what  conditions,  how 
HARPOON  seeker  characteristics  can  be  used  to  increase  the 
probability  of  acquiring  a  selected  target  in  a  formation, 
etc. 

c.  Tactical  procedures  —  the  means  by  which  a  com¬ 
mander  could  implement  tactical  concepts  (e.g.,  maneuver  so 
that  the  target  has  an  open-ocean  background). 

1304.  What  These  Elements  Mean.  In  the  OPTEVFOR  view,  tac¬ 
tics,  first  and  foremost,  is  a  way  of  thinking  —  a  thought 
process.  An  OPTEVFOR  Tactics  Guide  assists  a  fleet  user  in 
his  thinking  process,  by  providing  some  of  the  framework  for 
his  thinking.  The  Guide  does  not  present  dogma  or  cookbook 
style  do's  and  don'ts,  but  rather  is  a  thoughtful  treatment 
of  aspects  of  the  equipment  that  the  user  must  understand  if 
he  is  to  use  it  wisely.  It  tells  him  the  things  he  should 
be  thinking  about  in  making  his  decision  to  use  the  equip¬ 
ment  . 

1305.  Tips  on  Planning  or  Writing  an  OTG.  The  following 
tips  have  been  developed  through  experience  —  there  is  no 
significance  to  the  order  of  their  appearance. 

a.  Stay  alert  to  tactical  considerations  from  the  time 
you  first  hear  of  a  new  project  start. 

b.  Do  not  fill  an  OTG  with  "knobology." 

c.  Ask  yourself  what  you  would  need  to  know  if  you  were 
a  fleet  user  of  the  equipment. 

d.  Interface  with  the  fleet  before,  during,  and  after 
writing. 
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e.  Use  every  opportunity  to  take  part  in  fleet  operations 


f.  Define  the  level  of  fleet  user  you  are  going  to 
address  the  OTG  to  (i.e.,  weapons  officer,  CO,  aircraft 
commander,  task  group  commander,  etc.)* 

g.  Bounce  your  ideas  off  other  people  —  never  write  in 
isolation. 

h.  Bring  higher  levels  of  command  (Section  Head,  ACOS, 

02,  02T,  etc.)  into  the  discussion. 

i.  Don't  try  to  impress  the  reader  with  your  education. 
Keep  mathematics  out  of  the  OTG  as  much  as  possible. 

j.  Consult  other  communities. 

k.  Keep  the  OTG  as  concise  as  possible. 

l.  Do  a  lot  of  thinking.  OTGs  are  mostly  derived  from 
a  little  bit  of  testing,  plus  a  lot  of  thought  in  writing. 

m.  Use  diagrams  —  saves  words. 

n.  Keep  the  security  classification  as  low  as  possible. 

o.  When  designing  OT&E,  work  out  the  Test  Plan  to  maxi¬ 
mize  the  useful  operational  data  to  be  obtained. 

p.  When  designing  a  test  and  collecting  data,  remember 
that  the  subjective  impressions  of  those  carrying  out  the 
test  can  be  very  important  in  formulating  tactics  and  decid¬ 
ing  how  best  to  use  the  equipment  —  the  human  factor. 

q.  The  OTG  format  is  flexible  —  take  advantage  of 
this.  Make  the  product  readable.  Nobody  is  forced  to  read 
it  —  it  has  to  sell  itself  in  early  paragraphs. 

r.  It  is  better  to  explain  the  capabilities  and  limi¬ 
tations  of  the  equipment,  and  from  that  what  you  think  its 
tactical  philosophy  should  be,  them  to  give  detailed  tactical 
instructions . 

s.  Avoid  overemphasizing  equipment  weaknesses  —  strike 
the  proper  balance  between  capabilities  and  limitations. 

t.  Do  not  try  to  adapt  existing  tactics  to  new  equipment. 
Start  with  a  blank  sheet  of  paper.  Know  the  equipment,  and 
think  how  best  you  can  use  it.  Keep  in  mind  the  possibility 
of  improving  usage  as  a  result  of  tactical  feedback. 
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u.  Highlight  areas  where  operator  training  is  especially 
critical.  Where  appropriate,  suggest  methods  of  training. 

v.  Remember  that  OTGs  are  half  the  output  of  OT&E. 
(Evaluation  Reports  are  the  other  half. ) 

1306.  Publication  Dates  of  Tactics  Guides.  According  to 
OPTEVFORINST  3960. 2A,  when  a  Tactics  Guide  is  required  by 
the  TEMP,  it  will  be  published  within  120  calendar  days 
after  completion  of  project  operations. 

1307.  short  Titles.  A  sequential  numbering  system  is  used  for 
OTG  short  titles.  COMOPTEVFOR  Code  02T  (AUTOVON  690-5177) 
assigns  the  short  titles  for  all  OTGs. 

1308.  Cancellation  or  Review  Dates.  A  cancellation  or 
review  date  is  assigned  to  each  OTG,  to  assist  in  keeping 
tactical  information  current  and  non-obsolescent.  Code  02T 
advises  the  OTD  on  dates. 

1309.  Format  and  Content  of  OPTEVFOR  Tactics  Guides.  Use 
a  format  that  best  suits  the  material  you  are  presenting. 

A  sample  format  that  has  been  used  in  several  OTGs  is  con¬ 
tained  in  the  pages  following.  Use  it  if  you  like. 
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r  Downgrading  Statement* 

Classification* 

*li  applicable..  Vo  not  u*e  on 
UNCLASSIFIED  Tactic*  Gtu.dc* . 

T fie  iorm  oi  the  coven  i*  the.  *am( 
whether  the  guide,  i*  promulgated 
bu  a  VX  Squadron  or  by  COMOPTE i/Fd 
The  color  oi  the  cover  indicate* 
the  overall  cla**iiication  oi  the 
guide .  Irf  the  guide  i*  promulgai 
by  a  VX  Squadron,  the  applicable 
*quadron  *eal  will  appear  in  the 
upper  leit  corner. 
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DEPARTMENT  OF  THE  NAVY 

COMMANDER  OPERATIONAL  TEST  AND  EVALUATION  FORCE 
NORFOLK,  VIRGINIA  23511 


02T:cebjr 
3510 
Ser  XXX 
6  June  1981 


CLASSIFICATION* 

From:  Commander  Operational  Test  and  Evaluation  Force 

To:  Distribution 

Subj:  Tactics  Guide  XX-81  for  the  Mk  XX  Mod  0  Fire  Control  System 

(Report  Symbol  OPNAV  3960-13)  (*) 

{Thi*  * ample,  title,  indicate.*  that  0T- II  wa *  ju*t  completed 
on  the  Mfe  XX  Mod  0 ;  *taKt  the  title  with  " Preliminary "  on 
"Follow-on"  ii  0T- 1  ok  0T-T.ll  Ke*ulted  in  thi *  Guide.) 

Ref:  The  only  KefieKence  that  noamally  would  be  K^'uiKed  heae  i *  a 

pKeviou*  OPTEVFOR  Tactic *  Guide  be.-i.ng  *upeK*eded  ok  modified, 
ok  a  high-cla**iftication  *upplement  to  thi *  document . 

1.  (*)  This  OPTEVFOR  Tactics  Guide  contains  information  on  tactical 
employment  of  the  Mk  XX  Mod  0  FCS  (Fire  Control  System) ,  as  installed 

in  _  and  type  ships.  Section  1  of  this  Guide 

describes  the  Mk  XX  Mod  6  PCS  as  it  was  tested,  and  the  scope  of 
testing.  Section  2  discusses  the  tactical  capabilities/limitations 
of  the  FCS  considered  to  have  been  demonstrated  during  testing. 
Section  3  presents  recommended  tactics  for  employing  the  FCS. 

Section  4  describes  areas  that  warrant  further  investigation. 

2.  (*)  This  Tactics  Guide  summarizes,  for  early  fleet  use,  those 
tactical  considerations  OPTEVFOR  was  able  to  develop  during  OTSE 
(operational  test  and  evaluation) .  The  information  contained 
herein  is  considered  sound,  but  is  preliminary  in  nature  and 
therefore  subject  to  change.  Comments  on  the  tactics  and  procedures 
presented  are  invited  and  encouraged. 

3.  (*)  Navy  organizations  on  the  distribution  list  may  obtain 
additional  copies  of  this  document  from  Director,  Naval  Tactical 
Support  Activity,  in  accordance  with  OPNAVINST  5070.7.  All  other 
requests  for  copies  should  be  forwarded  to  (COMOPTEVFOR,  VX-1,  etc., 
as  appropriate) . 


DOWNGRADING  STATEMENT*  RequiKed  i£  not  on  coveK  *heet. 
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CLASSIFICATION* 


CLASSIFICATION* 

4.  (*)  The  following  pertains  to  this  Tactics  Guide: 

a.  TAC  D&E  Category  Code  (two-letter,  10  digit). 

b.  TAC  D&E  Number  (e.g.,  OTF  18-77,  OTF  3-78). 

c.  Appropriate  NWPs  (e.g.,  NWP  32,  NWP  55-2-P3) . 

d.  Cancellation  date. 


Distribution:  This  is  the  minimum  distxibution.  (fox 
pxogxams  sponsoxed  by  0P-0Z ,  see  OZT  fax 
minimum  distxibution.  )  Include  all  o{j  these 
in  all  Guides,  a*  indicated  below. 

CNO  (OP-090) 

(OP-96) 

(OP-095) 

(OP-953) 

(OP-098) 

(OP-981) 

(OP-983) 

(OP-O  )  (Pxogxam  Sponsox  (VMSO,  VCNO  ox  A CNO)) 

NAV  syscSm  (PM- _ )  (Cognizant  PM,  oi  thexe  i s  one) 

(_-__)  ( Cognizant  Veputy  ox  Assistant  Commandex ) 

( _ - _ )  (Cognizant  Acquisition  Uanagex) 

C INCLANTFLT 

CINCPACFLT 

CINCUSNAVEUR 

COMSECONDFLT 

COMTHIRDFLT 

COMSIXTHFLT 

COMSE VENTHF LT 

COM _ LANT 

COM  PAC 

PaxTZcipating  facet  unit* ,  including  the  pxoject  ship  (i&  any ) 

and  ISIC,  plus  all  potential  usex  ship ,  squaaxons ,  etc. 

COMOPTEVFOR 

DEPCOMOPTEVFORPAC 

NAVPGSCOL 

PRESNAVWARCOL 

CNET 

ONR 

CNA 

CO,  SWOSCOLSCOM 
NAVTAC INTEROP SUPP ACT 
DIRNAVTACSUPPACT  (50) 

2  CLASSIFICATION* 
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CLASSIFICATION* 


Contents  (*) 


Page 

Acronyms  and  Abbreviations  ii 

References  iii 

Section  1  —  Introduction 

101  —  System  Description  1-1 

102  —  Scope  of  Testing  1-1 

103  —  Limitations  to  Testing  1-1 

Section  2  —  Tactical  Considerations  2-1 

Section  3  —  Tactical  Applications  3-1 

Section  4  —  Areas  for  Further  Study  4-1 

Annex  A  —  In&tfiuctionA  &0K  Annex  Wfu.ti.ng 

A)  NOTE:  The  format  presented  is  only  an  example,  not  a  re¬ 
quired  format. 
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MICROCOPY  RESOLUTION  TEST  CHART 

n*TION«.  BUREAU  OF  STAN0AR0S4963-A 


CLASSIFICATIONS* 


Acronyms  and  Abbreviations  (*) 


Combat  Information  Center 


FCS  Fire  Control  System 

Acronyms  should  be  defined  (spelled  out)  on  the  first  occur¬ 
rence  in  the  text,  o-nd  listed  here.  Two  methods  of  spelling 
out  are  allowed  by  the  Navy  Correspondence  Manuals  i.e.t  CIC 
(Combat  Information  Center)  or  Combat  Information  Center 
(CIC).  The  former  method  ( acronym  first)  is  used  by  COMOPTEVFOR. 

Acronyms  for  naval  activities  included  in  the  Standard 
Naval  Distribution  List  (which  includes  almost  every  activity ) 
need  not  be  spelled  out  or  listed  on  the  acronym  page.  The 
Operational  Test  Director  is  not  precluded  from  spelling  out 
and  listing  such  acronymss  however3  if  readability  will  be 
improved  (e.g.t  acronyms  for  obscure  activities ) . 

Acronyms  which  are  defined  (spelled  out)  in  the  letter  need 
not  be  spelled  out  again  in  the  enclosures  except  on  this 
page. 


ii 
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References  (*) 

If  references  were  used  in  the  letter t  repeat  them  here  in  the  same 
order  in  which  they  appear  in  the  letter.  Follow  them  with  references 
mentioned  in  the  sections  to  follow ,  in  the  order  in  which  they  are 
first  mentioned  in  these  sections. 


ill 
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Section  1 
Introduction  (*) 


101.  (*)  System  Description 

The  purpose  of  this  paragraph  is  to  provide  a  sufficiently  detailed 
description  of  the  system  that  subsequent  discussions  of  its  capabilities/ 
limitations /employment  are  readily  understood.  If  the  system  being 
discussed  is  completely  new  and  therefore  not  well  known  in  the  fleet, 
this  paragraph  may  be  quite  lengthy.  If*  on  the  other  hand ,  the  system 
is  an  improved  version  of  an  older  system ,  this  paragraph  need  only 
address  the  improvements  and  can  be  relatively  short.  The  use  of  photo¬ 
graphs ,  diagrams,  and  tables  for  conciseness  and  clarity  is  encouraged. 

Within  this  paragraph,  describe  any  ways  the  system  tested  is  known  to 
differ  from  the  system  to  be  installed  in  the  fleet.  These  differences 
include  system  differences  per  se,  and  differences  in  the  way  the  system 
will  be  installed  (for  instance,  antenna  location).  Be  as  operationally 
specific  as  possible  (for  instance,  " the  system  tested  did  not  receive 
inputs  from  the  doppler  radar,  etc."),  as  opposed  to  developmentally 
general  (for  instance,  " the  system  tested  was  a  prototype") .  This  type 
statement  conveys  little  useful  information  to  the  operational  commander. 


102.  (*)  Scope  of  Testing 

The  purpose  of  this  paragraph  is  to  describe  what  was  done  which  lead  to 
the  tactical  employment  considerations  discussed  later.  The  object  is 
to  present,  as  clearly  as  possible,  a  stannary  of  the  testing,  so  the 
reader  can  decide  for  himself  how  much  confidence  to  place  in  our  find¬ 
ings  and  recommendations.  The  important  elements  of  this  paragraph  are 
the  ship,  aircraft,  etc.,  in  which  the  system  was  installed,  and  the 
scenarios  in  which  the  system  was  exercised,  together  with  a  summary  of 
the  amount  of  time  (and  weapons  delivered,  etc.)  the  system  was  exer¬ 
cised.  Do  not  include  material  of  no  interest  to  operational  commanders, 
such  as  listings  of  suitability  tests.  Do  include  pertinent  information 
on  weather  conditions  during  the  testing,  and  level  and  type  of  enemy 
threat  the  system  was  employed  against. 

If  simulations  were  employed  in  the  testing,  they  should  be  mentioned 
specifically.  Simulations  include  U.S.  -built  versions  of  threat  emit¬ 
ters,  and  computer  simulations  of  missile  intercepts. 

103.  (*)  Limitations  to  Testing 


Identify  here  the  aspects  of  the  new  weapons  system  that  were  not 
adequately  tested.  Inadequate  testing  is  defined  to  include  a  total 
absence  of  testing,  and  testing  whose  results  are  suspect  because  of 
limited  data,  unrepresentative  pretest  preparation ,  etc.  The  purpose  of 
this  paragraph  is  to  flag  for  the  reader  those  aspects  of  the  system 
that  we’re  not  sure  we  have  a  complete  handle  on  —  to  avoid  mis¬ 
leading  him.  _ 
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Section  2 

Tactical  Considerations  (*) 

This  section  diecussee  the  capabilities  and  limitations  of  the  system 
that  were  determined  during  testing,  and  on  which  recommended  tactics 
were  based.  The  purpose  is  to  identify  known  system  elements ,  so  that 
they  need  not  be  reestablished  by  fleet  units  investigating  different 
tactics  or  different  scenarios.  These  known  system  elements  are  those 
operationally  interesting  parameters  that  have  been  sufficiently  defined 
for  reasonable  confidence.  They  include  such  things  as  target  acquisi¬ 
tion  range  as  functions  of  target  size,  geometry ,  dynamics,  etc.  They 
include  (and  these  are  very  important)  negative  system  elements,  such  as 
a  DECM  system's  inability  to  counter  LOW  BLOW.  This  section  contains, 
then,  a  listing  of  the  system's  tactical  capabilities  and  its  tactical 
limitations  that  form  the  basis  of  any  discussion  of  tactical  employ¬ 
ment. 

The  organization  of  this  section  should  present  the  facts  in  the  most 
understandable  manner.  In  some  cases  this  section  will  best  be  organ¬ 
ized  by  addressing  individual  missions  in  which  the  system  will  be 
employed.  In  other  cases  it  will  best  be  organized  by  discussing 
system  modes  of  operation.  Still  others  will  best  be  organized  by 
threat  categories.  No  rules  are  established,  except  the  standard  one  to 
strive  for  accuracy,  readability ,  clarity,  and  brevity,  in  that  order. 
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Section  3 

Tactical  Applications  (*) 

This  section  provides  guidance  on  the  tactics  to  use  with  a 
new  weapons  system.  This  guidance  may  take  many  forms.  For 
a  towed  array  it  might  be  an  operating  guideline  addressing 
questions  such  as  the  depth  to  operate  as  a  function  of 
layer,  or  bearing  resolution  procedures .  For  a  projectile 
or  fuze,  it  might  be  a  decision  matrix  of  pro jectile/ fuze 
combinations  for  different  targets.  Realistic  operational 
situations  might  be  posed  (XJZ  missile  ready  to  launch , 
enemy  deploys  chaff),  and  our  guidance  specifies  the  beet 
tactic  in  response  (cheok  fire,  fire  salvo  of  three,  etc.). 

Organize  the  section  as  logically  as  possible.  Consider 
organizing  it  to  parallel  Section  2. 
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Section  4 

Areas  for  Further  Study  (*) 

This  section  identifies  areas  that  warrant  further  investi¬ 
gation.  Some  of  these  areas  may  follow  from  the  discussion 
in  paragraph  10Zt  Limitations  to  Testing .  Others  may  be 
suggested  by  possible  changes  in  the  threat ,  or  by  possible 
other  uses  of  the  equipment. 
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Annex  A 


Instructions  for  Annex  Writir 


1.  Annexes  present  pertinent  material  not  appropriate  for 
inclusion  in  the  body  of  the  Guide t  because  its  length  or 
detail  would  destroy  the  continuity  of  the  text.  Such 
material  would  be  detailed  tabulations  of  equipment  charac- 
teristic8t  or  voluminous  sketches  of  run  geometries . 


2.  If  annexes  are  used ,  they  must  be  referred  to  in  the 
appropriate  place ( e)  in  the  text  of  the  main  body  of  the 
Guide ,  and  listed  on  the  contents  page. 
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Section  14 

Oral  Presentations  On  OT&E 

1401.  General.  OT&E  briefings  are  like  any  other  Navy 
briefings  —  they  cover  the  facts  as  we  know  them  in  a 
logical,  concise  fashion.  They  are  usually  conducted  at  the 
Flag  level,  and  are  often  called  for  ahead  of  their  sched¬ 
uled  dates,  with  little  warning.  Some  specific  guidance  on 
OT&E-peculiar  aspects  is  provided  below. 

1402.  Before  Operational  Testing.  Frequently  an  OTD  or  OTC 
will  be  called  upon  to  brief  the OT&E  plan  to  decision 
makers,  usually  to  show  them  how  OT&E  will  address  issues 
critical  to  the  next  program  decision.  If  you  find  yourself 
in  this  position,  first  and  most  important  —  make  sure  you 
understand  what  these  issues  are.  Then,  for  your  briefing: 

a.  State  these  issues  —  but  only  those  that  can  be 
resolved  through  OT&E. 

b.  Then,  as  simply  and  clearly  as  possible,  shown  how 
the  OT&E  plan  addresses  them. 

c.  Where  appropriate  and  possible,  give  the  audience  a 
feel  for  how  much  confidence  our  testing  should  give  us. 

For  instance  —  if  KTBF  is  a  critical  issue  —  show  what  the 
MTBF  would  be  at  the  80%  confidence  level  if  the  equipment 
operates  for  the  planned  duration  of  OPEVAL  with  only  1  or  2 
failures.  By  the  way  —  use  numbers  that  assume  that  the 
equipment  works  well  —  don't  presuppose  that  the  equipment's 
no  good. 


1403.  When  Operational  Testing  is  Accomplished, 
on  OT&E  results  are  best  structured  as  follows: 

a.  Objectives  of  the  phase  completed. 


Briefings 


b.  Brief  summary  of  testing  (number  of  rounds  fired, 
etc.),  and  major  limitations  to  the  evaluation. 

c.  Results  -  keyed  to  objectives. 

d.  Operational  considerations  (particularly  as  regards 
interpreting  results ) . 

e.  Conclusions  on  operational  effectiveness  and  opera¬ 
tional  suitability. 
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f.  Recommendations. 


R) 


If  analysis  is  incomplete,  say  so.  If  conclusions  are  ten¬ 
tative,  say  so.  Etc. 

1404.  Programmatic  and  Future  Test  Issues.  If  you  brief  on 
these  and  related  subjects,  remember  that  you  are  briefing 
from  an  OT&E  viewpoint.  Don’t  include  things  not  within  our 
area  of  responsibility  (e.g.,  system  cost  versus  cost  of  a 
similar  system). 


1405. 


>ical  OT&E  Brie  fine 


lirements 


a.  Periodic  Program  Reviews  (OPNAV/SYSCOM/Labs) . 


b.  Milestone  II  Preview. 


c.  Milestone  II. 


d.  Milestone  III  Previews  (Pre-CEB,  CEB,  Pre-DNSARC, 
DNSARC,  DSARC,  DDTE  Review). 

e.  Milestone  III. 


1406.  Sample  CEB-Type  Briefing.  In  the  following  pages  is 
an  example  of  a  management-level  briefing  on  results  of  an 
OPEVAL.  The  management  decision  being  considered  is  the 
ASU/production  decision.  Note  that  this  sample  briefing  is 
geared  to  an  audience  interested  primarily  in  top-level 
results,  conclusions,  and  recommendations  — not  in  details. 
Details  are  necessary,  however: 

a.  If  we  recommend  against  the  system.  The  briefing 
must  fully  substantiate  negative  conclusions  and  recommen¬ 
dations  . 


b.  If  the  briefing  is  to  DDTE  —  this  briefing  is  much 
more  test-oriented  than  the  decision-oriented  briefing  of 
the  sample. 


(Change  1) 
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We  are  now  ready  to  address  the  major  test  results  —  keyed  to  objectives. 
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Section  15 


Creation  and  Maintenance  of  an  OTP  Journal 


1501.  Introduction.  Each  QTD  should  maintain  a  chronologi¬ 
cal  record  of  his  project.  Hie  purpose  of  this  record  is 
multiple;  for  instance,  it  provides  a  history  for  your 
replacement  in  the  event  you  are  transferred;  it  may  enable 
you  to  answer  new  questions  about  an  old  test;  it  can  serve 
as  substantiating  data  if  events,  agreements,  etc.  are  later 
questioned.  (It  may  be  the  sole  record  of  something  that 
later  becomes  important.)  This  record  may  exist  in  several 
forms;  loose-leaf  notebooks,  steno  pads,  memos  for  the 
record,  cassette  recordings,  etc.  Collectively  they  are 
called  an  OTD  Journal.  If  an  individual  OTD  Journal  consists 
of  a  combination  of  steno  pads,  recordings,  etc.,  one  docu¬ 
ment  (the  Master)  should  maintain  the  overall  chronology, 
and  should  reference  individual  steno  pads,  recordings,  etc. 
for  details,  where  appropriate. 

1502.  Content .  The  OTD  Journal  records  for  possible  later 
use  everything  the  OTD  considers  of  significance  in  his 
program.  While  each  OTD  must  use  his  own  judgement  when 
deciding  what  is  significant,  it  is  better  to  record  too 
much  than  too  little.  And  it  is  better  to  record  as  soon  as 
an  event  occurs,  rather  than  to  wait  until  later  and  risk 
forgetting.  Among  the  things  which  may  have  significance 
are: 

a.  Funding  requirements/transactions  for  OT&E. 

b.  Agreements  made  at  meetings  or  over  the  phone  regarding 
future  testing. 

c.  Summaries  of  program  meetings  and  conferences, 
including  attendees,  areas  of  discussion,  and  stands  taken 
by  the  various  players. 

d.  Mention  of  working  drafts,  etc.  exchanged  between 
the  OTD  and  other  program  individuals  or  offices,  with 
notations  indicating  where  copies  may  be  found  in  the  OTD's 
files . 


e.  Notations  summarizing  oral  business  contacts  with 
individuals  associated  with  the  program  (CNO,  NAVMAT,  SYSCOM, 
Labs,  other  Services,  DDR&E,  contractors,  etc.)  together 
with  their  codes,  symbols,  phone  numbers,  etc. 
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£.  Mention  of  receipt  of  incoming  program  messages, 
letters,  data  packages,  etc.,  together  with  their  storage 
locations. 

g.  An  on-scene  record  of  testing  (see  paragraph  1503 
below) . 


h.  A  record  of  drafts  (messages,  reports,  etc.)  pre¬ 
pared  for  higher- level  review  and  approval  (draft  completion 
dates,  cut-board  dates,  significant  events  in  the  review 
process,  approval  dates,  etc.). 

i .  Identification  (by  date/time  group  or  serial  number 
and  date)  of  outgoing  program  documentation,  with  primary 
addressee  and  storage  location. 

j.  Significant  program  information  (funding  changes, 
schedule  slippages,  PASU,  etc.),  together  with  the  source  of 
the  information. 

k.  The  line  of  reasoning  that  led  you  to  take  a  particular 
stand  on  an  issue,  or  that  caused  you  to  select  certain 
parameters,  etc.  This  may  be  of  critical  importance  to  your 
replacement  who  is  trying  to  figure  out  why  you  set  things 

up  the  way  you  did. 

1003.  On-Scene  Record  of  Testing.  A  running  account  of 
testing  is  an  important  part  of  an  OTD  Journal.  In  many 
cases  this  account  is  best  made  on  a  cassette  recorder  as 
the  operation  progresses.  (Don't  forget  extra  cassettes  and 
batteries — and  get  somebody  assigned  to  transcribe  for  you.) 

In  any  event,  its  purpose  is  to  describe  the  way  the  testing 
actually  occurred;  what  happened,  when,  and  who  (what)  was 
involved.  It  identifies  the  operation  (by  run  number, 
etc.),  and  provides  a  running  time-correlated  commentary  to 
the  end  of  the  exercise.  Particular  attention  is  on  record¬ 
ing  unusual  events  (breakdowns  in  communications,  intruders 
in  the  area,  etc.).  Differences  between  actual  and  planned 
scenarios  are  noted  and  explained.  The  OTD's  impressions, 
qualitative  assessments  of  performance,  and  any  other  infor¬ 
mation  which  later  might  help  him  reconstruct  the  testing, 
are  recorded.  Keep  in  mind  that  an  OTD  Journal  is  your 
document,  to  help  you  (and  your  successor).  It's  like  a 
computer  —  you  only  get  out  what  you  put  in. 

1504.  Pack-up  Kit.  Check  with  our  Supply  regarding  instru¬ 
ments  ancT  other  aids  to  you  when  you  go  off  on  an  operation. 


1505.  Retention  of  Test-Related  Information 


a.  The  OTD's  Journal,  together  with  the  documents,  data 
packages,  etc.  that  are  referred  to  in  the  Journal  should  be 
retained  by  the  OTD  as  long  as  the  CNO-assigned  T&E  number 
remains  active. 

b.  When  the  CNO-assigned  T&E  number  is  retired,  the  OTD 
should  take  the  following  actions: 

(1)  Make  a  list  of  all  project  documents  originated 
outside  the  Force,  by  classification,  originator,  type, 
identifier,  data,  and  subject  —  the  same  way  they  would  be 
listed  in  a  "Reference"  page  of  an  Evaluation  Report.  Label 
this  list  "Project  Documents  Not  Retained"  and  destroy  the 
basic  documents  in  accordance  with  established  administrative 
and  security  procedures. 

(2)  Make  up  a  retention  package  containing  the  OTD 
Journals,  the  list  marked  "Project  Documents  Not  Retained," 
and  single  copies  of  significant  documents  originating  from 
within  the  Force  ( including  data  packages ) .  Mark  the 
retention  package  "T&E  Number..."  and  forward  it  (with 
inventory)  to  COMOPTEVFOR  (Admin)  for  storage,  and  notify 
the  DCOS  of  this  action. 
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Section  16 


OTC/OTD  Relationships  With  Others 

1601.  Introduction.  In  business  dealings  with  other  agencies/ 
organizations,  the OTC  or  OTD  acts  as  a  representative  of 
COMOPTEVFOR.  In  these  dealings  he  presents  the  COMOPTEVFOR 
viewpoint,  as  he  knows  it,  based  on  written  and  oral  guidance 
and  good  judgement.  If  that  viewpoint  is  not  accepted,  he 
retires  politely  and  initiates  action  to  get  this  viewpoint 
put  into  writing  for  COMOPTEVFOR  action. 

1602.  With  CNO  Staff.  These  dealings  will  usually  fall 
into  the  category  of advising  CNO  on  the  adequacy  of  planned 
T&E.  Often  they  will  involve  a  disagreement  with  the  DA 
regarding  the  time  for,  or  cost  of,  proper  OT&E.  Remember 
that  OPTEVFOR  is  usually  the  only  advocate  of  proper  OT&E 
below  the  CNO  level.  Concentrate  on  making  OT&E  advocates 
within  the  CNO  staff  —  at  least  advocates  for  your  own 
OT&E.  Convince  them  of  the  real  need  for  this  OT&E;  show 
them  how  a  relatively  small  outlay  in  targets,  etc.,  can 
result  in  profound  savings  later  on  and  in  better  systems  in 
the  fleet,  earlier.  Remember  that  most  of  these  officers 
don't  make  decisions;  their  job  is  to  convince  those  who  do. 
Provide  them  with  convincing  arguments;  don't  just  tell  them 
that's  the  way  COMOPTEVFOR  wants  it. 

1603.  With  the  DA 

a.  It  helps  if  you  can  convince  the  DA  that  we're  not 
out  to  shoot  his  program  down.  In  every  program  there  will 
be  disagreement  regarding  requirements  for  T&E;  if  the  DA  is 
at  least  aware  that  we're  on  the  side  of  fielding  good 
equipment,  these  disagreements  will  be  resolved  with  less 
noise.  The  first  disagreement  will  probably  be  associated 
with  program  structure;  how  T&E  supports  the  milestones. 

Present  the  COMOPTEVFOR  position  (see  Section  6),  explain 
this  position,  and  if  possible  cite  examples  of  programs 
that  suffered  because  of  poor  structure.  As  much  as  possible 
keep  the  program  moving  in  the  right  direction  through 
working  meetings,  informal  exchanges  of  working  drafts,  etc. 

But  if  the  program  stops  or  is  moving  uncontrollably  in  the 
wrong  direction,  go  formal  through  your  own  chain  of  command. 

If  you  are  put  down  because  you're  not  a  technical  expert, 
remember  that  your  technical  qualifications  are  immaterial. 
You're  an  operational  expert;  no  one  else  connected  with  the 
program  may  ever  have  been  to  sea.  You  and  the  people  in 
the  project  office  are  not  debating  opponents;  you  are 
allies  in  getting  good  equipment  in  the  fleet. 


b.  In  some  programs  it  may  be  convenient  (or  absolutely 
necessary)  to  use  DA  field  agencies  to  get  OT&E  data  reduced 
and,  to  some  degree,  analyzed.  In  these  situations  it i  is 
mandatory  that  these  people  be  under  the  operational  control 
of  COMOPTEVFOR  (represented  by  the  OTC/OTD/Program  Analyst) 
while  they  are  working  on  OT&E  data.  Their  work  is  defined 
in  advance,  and  their  results  are  furnished  only  to  COMOP¬ 
TEVFOR,  unless  COMOPTEVFOR  has  specifically  approved  a  wider 
distribution. 

1604.  With  the  Manufacturer.  You  will  be  involved  with  him 
at  meetings,  program  reiews,  critical  design  reviews,  visits 
to  the  plant,  and  during  factory  testing.  Deal  with  him  as 
much  as  is  necessary  to  get  our  job  done.  But  avoid  any 
association  with  him  which  could  possibly  compromise  your 
(and  therefore  COMOPTEVFOR ' s )  independence  and  integrity. 

If  you  need  to  spend  all  day  at  his  plant,  have  lunch  there 
but  pay  for  it.  Don't  go  out  to  dinner  with  him.  At  the 
same  time,  try  not  to  leave  any  impression  of  hostility  or 
mistrust.  Be  professional  and  remember  that  you're  a 
representative  of  the  Navy's  independent  OT&E  organization. 
Maintain  not  only  the  independence,  but  also  the  appearance 
thereof.  Also,  remember  that  in  dealings  with  the  manu¬ 
facturer  you're  working  as  part  of  the  Navy's  development 
team,  and  that  visits  to  the  manufacturer  are  arranged  through 
and  with  the  permission  of  the  DA.  Go  in  uniform  and  check 
in  with  the  NAVPRO. 

1605.  With  Cognizant  Navy  Field  Activities.  While  these 
are  closely  related  to  the  Washington  DA,  you  will  usually 
find  them  more  receptive  to  your  views  than  is  the  DA.  This 
is  because  they  can  veiw  T&E  requirements  (including  OT&E) 
from  a  technical  viewpoint  not  obscured  by  costs  or  schedules 
(as  a  good  DA  must).  In  spite  of  this  difference,  the 
guidance  of  paragraph  1603  is  basically  applicable. 
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Section  17 


COMOPTEVFOR  in  JOT&E  (Joint  OT&E) 

1701.  Background.  For  weapon  system  development/acquisi¬ 
tion  programs  established  by  joint  agreement  between  two  or 
more  services,  the  T&E  conducted  during  development  is  re¬ 
ferred  to  as  JT&E  (joint  test  and  evaluation).  The  testing 
for  operational  evaluation  is  referred  to  as  JOT&E.  JT&E 
programs  may  also  be  initiated  by  DDTE  in  accordance  with 
DOD  Directive  5000.3. 

1702.  COMOPTEVFOR  I nvo 1 vemen t .  The  extent  of  COMOPTEVFOR 
participation  in  JOT&E  is  determined  on  a  case-by-case 
basis.  OTD/OTC  responsibilities  regarding  this  partici¬ 
pation  differ  according  to  whether  or  not  the  Navy  is  lead 
service  in  the  development/testing. 

1703.  Navy  Lead  Service.  If  the  Navy  is  lead,  the  provi¬ 
sions  of  OPNAVINST  3960.10  apply  to  the  JOT&E,  and  COMOPTEV¬ 
FOR  performs  essentially  the  same  functions  as  in  ordinary 
OT&E,  with  the  following  modifications: 

a.  All  planning  is  coordinated  with  the  other  services. 

b.  The  COMOPTEVFOR  Test  Plan  may  include  testing  to  be 
performed  by  the  other  service.  If  this  is  the  case,  differ¬ 
ences  in  perferred  format,  working,  etc.  may  dictate  that 
the  other  service's  verbatim  input  be  included  as  a  separate 
section  or  annex  in  the  Test  Flan. 

c.  If  Test  Plans  do  include  both  services'  testing, 
there  will  probably  be  a  joint  sign-off  on  the  letter  of 
promulgation  (and  an  associated  joint  review  process).  This 
requirement  should  be  taken  into  account  in  estimating  pub¬ 
lication  lead  time,  etc.  (Note:  this  can  apply  equally  to 
other  JOT&E  documents  —  reports,  for  example.  The  ground 
rules  regarding  who  prepares  and  signs  what  require  early 
resolution  —  again  on  a  case-by-case  basis.) 

d.  Whether  or  not  combined  oi  separate  Test  Plans  are 
issued,  it  is  usually  extremely  important  that  the  methods 
by  which  each  service  evaluates  the  same  thing  are  identi¬ 
cal  .  That  is,  in  reliability  calculations,  each  service 
includes  the  same  failure  types,  uses  the  same  criteria  for 
no-tests,  etc.  These  matters  must  be  understood  and  agreed 
to  in  advance  of  testing.  Working  out  these  agreements  is  a 
major  function  of  the  OTD  in  JOT&E  planning. 
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1704.  Other  Lead  Service.  When  another  service  is  lead, 
OPNAVINST  3960.10  does  not  apply  directly.  However,  because 
DOD  Directive  5000.3  is  applicable  to  all  services,  the 
philosophy  of  OPNAVINST  3960.10  does  apply,  and  the  OTD's 
functions  will  be  similar  to  those  described  in  paragraph 
1703. 

1705.  Non-Acquisition  JOT&E.  Non-acquisition  JT&E  (concept 
testing)  is  directed  by  DDTE,  who  also  budgets  and  funds  it. 
If  the  Navy  is  involved,  OP-983  normally  asks  COMOPTEVFOR  to 
review  the  proposed  testing,  to  see  if  it  is  "operational." 
If  it  appears  to  be  in  our  bailiwick,  we  assign  an  OTD.  He 
then  will  work  under  a  Joint  Test  Directorate  whose  Navy 
Deputy  is  usually  appointed  from  an  appropriate  operational 
command.  The  OTD*s  test  function  is  similar  to  that  he 
would  have  in  normal  operational  testing.  The  evaluation 
function  may  be  quite  different,  however,  because  the  Joint 
Test  Directorate  (not  COMOPTEVFOR)  reports  results  to  DDTE 

( for  approval ) . 

1706.  Assignment  Within  COMOPTEVFOR.  Policy  and  procedures 
for  assigning  responsibilities  for  planning,  prosecuting, 
monitoring,  and  reporting  JOT&E  is  contained  in  COMOPTEVFOR- 
INST  3930.6.  For  assistance  in  JOT&E  matters,  contact  Code 
02D. 


(Change  1) 
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Section  18 


Glossary 

ACAT.  Acquisition  category.  See  OPNAVINST  3960.10  for 
types  and  criteria. 

ASU.  Approval  for  service  use.  See  OPNAVINST  4720.9. 

Availability.  A  measure  of  the  degree  to  which  an  item  is  (R 
in  an  operable  and  commitable  state  at  the  start  of  a  mis¬ 
sion  when  the  mission  is  called  for  at  an  unknown  (random) 
time.  In  OT&E,  A  (operational  availability)  is  the  usual 
measure.  (See  Operational  Availability. ) 

Compatibility .  One  of  the  elements  of  operational  suitabil¬ 
ity!  The  capability  of  a  system  (or  subsystem)  to  operate 
in  its  intended  environment  without  adverse  effects  to  or 
from  other  systems.  Includes  effects  from  vibration,  radia¬ 
tion,  power  fluctuations,  etc. 

Critical  Issues.  Those  aspects  of  a  system's  capability,  (A 

either  operational,  technical,  or  other,  that  must  be  ques¬ 
tioned  before  a  system's  overall  worth  can  be  estimated,  and 
that  are  of  primary  importance  to  the  decision  authority  in 
reaching  a  decision  to  allow  the  system  to  advance  into  the 
next  acquisition  phase. 

Critical  Failure.  One  that  prevents  the  system  from  perform¬ 
ing  its  mission.  (Compare  Major  Failure  and  Minor  Failure.) 

DA.  Developing  Agency  (usually  a  SYSCOM). 

DDTE.  Director  Defense  Test  and  Evaluation.  According  to  (A 
DOD  Directive  5000.3,  DDTE  has  overall  responsibility  for 
T&E  matters  within  the  DOD. 

DSARC.  The  Defense  Systems  Acquisition  Review  Council  is  an 
advisory  body  to  the  Secretary  of  Defense  on  the  acquisition 
of  major  defense  systems.  Among  other  duties,  the  DSARC 
makes  recommendations  to  SECDEF  at  major  program  milestones, 
based  on  its  review  of  program  progress.  See  DOD  Directive 
5000.2. 

DT&E.  Development  test  and  evaluation.  See  Section  2  for 
comparison  with  OT&E. 

Evaluation  Criteria.  Standards  by  which  achievement  of  re-  (a 
quired  operational  effectiveness/suitability  characteristics, 
or  resolution  of  technical  or  operational  issues  may  be 
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Logistic  Supportability.  The  degree  to  which  the  planned  logistics 
(including  rest  equipment,  spares  and  repair  parts,  technical  data, 
support  facilities,  and  training)  and  manpower  meet  system  availability 
and  wartime  usage  reguirements . 

Maintainability.  The  capability  of  an  item  to  be  retained  in  or 
restored  to  specified  conditions  when  maintenance  is  performed  by 
personnel  having  specified  skill  levels,  using  prescribed  procedures 
and  resources,  at  each  prescribed  level  of  maintenance  and  repair. 

MTFL,  MTTR,  and  MSI  are  frequently  calculated  in  maintainability 
evaluations . 

Major  Failure.  One  that  causes  the  system  to  lose  some  operational 
capability,  and  degrades  mission  accomplishment.  If  detected  before 
the  mission,  would  probably  be  mission-aborting.  (Compare  Critical 
Failure  and  Minor  Failure. ) 

MFHBF/MTBF .  See  Reliability. 

Milestone  0  Decision.  The  decision  to  investigate  ways  to  acquire 
a  new  capability  to  meet  a  valid  need  essential  to  the  mission. 

Milestone  I  Decision.  The  decision  to  explore  a  concept  for  mission 
accompl i shment . 

Milestone  II  Decision.  The  decision  to  begin  engineering  development 
of  a  concept. 

Milestone  III  Decision.  The  decision  to  produce  a  system. 

Minor  Failure.  Affects  performance  but  can  be  worked  around  to  avoid 
impacting  the  mission.  (Compare  Critical  Failure  and  Major  Failure.) 

MIP  (Master  Information  Paper).  A  document  used  in  support  of  the 
proposed  program/budget.  See OPNAVINST  5260 . 1 . 

Mission  Reliability.  See  Reliability. 

MSI  {Maintenance  Support  Index).  The  ratio  of  total  man-hours 
required  for  maintenance  (preventive  plus  corrective)  to  the  total 
operating  (up)  time.  Frequently  computed  as  part  of  Test  S-2, 
Maintainability . 

MTFL.  Mean  time  to  fault-locate.  The  total  fault-location  time 
divided  by  the  number  of  critical/major  failures.  Frequently 
computed  as  part  of  Test  S-2,  Maintainability . 


MTTR.  Mean  time  to  repair.  Frequently  computed  as  part  of  Test  S-2, 
Maintainability,  and  usually  computed  as  MTTR  (geometric  MTTR). 

MTTR  is  the  antilog  of  the  quotient  obtainedgby  dividing  the  sum  of 
the  fogs  of  the  individual  critical/major  repair  times  by  the  number 
of  critical/major  repair  actions. 

New  Weapons  Systems .  Weapons  systems  whose  characteristics  have  been 
defined  explicitly,  but  that  have  not  yet  entered  the  inventory  in 
appreciable  numbers.  OT&E  of  new  weapons  systems  spans  OT-O  through 
OT-II.  COMOPTEVFOR  is  responsible  for  developing  tactics  for  new 
weapons  systems.  (Compare  Future  Weapons  Systems  and  Existing  Weapons 
Systems . ) 

Operability.  See  "Interoperability." 

Operational  Availability.  (See  Availability  for  basic  definition.) 

Aq  is  measured  in  two  different  ways  according  to  equipment  usage. 

1.  For  start-stop  type  equipment  (e.g.,  a  diving  suit's  life 
support  system,  a  radio  transmitter  for  an  SSBN,  a  machine  gun),  A 
is  the  ratio  of  successful  starts  (turn-ons,  actuations)  to  attempted 
starts . 


2.  For  more-or-less  continuously  operated  equipment  (e.g.,  search 
radars,  passive  sonars),  A  is  the  ratio  of  up  time  (performing  its  * 
mission  or  in  alert,  capable  of  performing  its  mission)  to  the  sum  of  - 
up  time  plus  down  time  (down  for  preventive/corrective  maintenance, 
awaiting  repair  parts,  etc.). 

Operational  Effectiveness .  Fundamentally,  the  capability  of  a  system, 
when  it  is  operating  the  way  it  is  supposed  to  (e.g.,  not  broken),  to 
perform  a  necessary  miss ion/f unction.  Operation  in  the  presence  of 
enemy  action  is  assumed;  hence  Survivability  and  Vulnerability  are 
treated  as  part  of  operational  effectiveness.  See  DOD  Directive  5000.3 
for  a  formal  definition. 

Operational  Suitability.  Fundamentally,  the  likelihood  that  in  the 
intended  operational  environment  the  system  will  perform  the  way  it 
is  supposed  to.  See  DOD  Directive  5000.3  for  a  formal  definition. 

OPEVAL.  Operational  evaluation.  The  last  phase  of  IOT&E. 

OPNAVINST  3960.10.  The  fundamental  Navy  instruction  on  T&E. 

OPSEC  (Operations  Security) .  See  paragraph  1006  for  a  discussion 
of  this  subject. 
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OT&E .  Operational  test  and  evaluation. 


PASU.  Provisional  ASU.  See  OPNAVINST  4720.9. 

PAT&E.  Production  acceptance  test  and  evaluation.  See 
OPNAVINST  3960.10. 

Quick-Look  Report.  An  informal,  usually  mess age- format, 
evaluation  report  published  by  COMOPTEVFOR .  Always  super¬ 
seded  by  a  formal  evaluation  report. 

RDT&E.  Research,  development,  test,  and  evaluation.  See 
NAVSO  P-2457  (RDT&E  Management  Guide). 

Reliability.  The  duration  or  probability  of  failure-free 
performance  under  stated  conditions.  In  OT&E,  reliability 
is  usually  reported  in  one  of  two  ways: 

1.  Mission  Reliability.  For  equipment  operated  only 
during  a  relatively  short-duration  mission  (as  opposed  to 
equipment  operated  more-or-less  continuously),  the  probabil¬ 
ity  of  completing  the  mission  without  critical  or  major  fail 
ure.  Frequently  expressed  as  exp  (-t/MTBF) ,  where  t  is  mis¬ 
sion  duration  and  MTBF  is  as  defined  below. 

2.  MTBF  (Mean  Time  Between  Failures).  For  more-or-less 
continuously  operated  equipment,  the  ratio  of  total  operat¬ 
ing  time  to  the  sum  of  critical  and  major  failures.  Some¬ 
times  modified  to  MFHBF  (mean  flight  hours  between  failures) 

Required  Operational  Characteristics .  System  parameters 
that  are  primary  indicators  of  the  system's  capability  to  be 
eiv.pl oyed  to  perform  the  required  mission  functions,  and  to 
be  supported. 

Required  Technical  Characteristics .  System  parameters  se¬ 
lected  as  primary  indicators  of  achievement  of  engineering 
goals.  These  may  not  be  direct  measures  of,  but  should  al¬ 
ways  relate  to  the  system’s  capability  to  perform  the  re¬ 
quired  mission  functions,  and  to  be  supported. 

Standardized  S-Tests.  In  OPTEVFOR  Test  Plans  the  following 
standardized  S-Tests  address  the  major  elements  of  opera¬ 
tional  suitability.  (Others  may  be  added,  as  appropriate.) 

Test  s-1.  Reliability 

Test  S-2 ,  Maintainability 
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Test  S-3 ,  Availability 

Test  S-4,  Logistics  Supportability 

Test  S-5,  Compatibility 

Test  S-6,  Interoperability 

Test  S-7,  Training  Requirements 

Survivability .  The  degree  to  which  a  system  is  able  to 
avoid  or  withstand  a  hostile  environment  without  suffering 
an  abortive  impairment  of  its  capability  to  accomplish  its 
designated  mission. 


TAC  D&E.  Tactical  development  and  evaluation  organizations. 
See  Existing  Weapons  Systems. 


Thresholds .  See  Evaluation  Criteria. 


T&E .  Test  and  evaluation.  See  OPNAVINST  3960.10. 


TEMP.  Test  and  Evaluation  Master  Plan.  The  controlling 
document  for  all  T&E.  See  OPNAVINST  3960.10  and  DOD  Direc¬ 
tive  5000.3  for  format  and  content. 


Vulnerability.  For  weapon  system  acquisition  decisions, 
three  considerations  are  critical  in  assessing  system  vul¬ 
nerability:  susceptibility  —  a  system  limitation  or  weak¬ 
ness  (may  not  be  exploitable);  accessibility  —  the  openness 
of  a  system  to  exploitation  by  a  countermeasures  technique; 
and  feasibility  —  the  practicality  and  probability  of  an 
adversary  exploiting  a  susceptibility  in  combat. 


(Change  1) 
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Annex  A 


Tips  on  Preparing  Correspondence 
A101 .  Fundamentals  of  Writing  OPTEVFOR  Correspondence 

a.  Logic .  The  fundamental  requirement  in  good  writing 
is  logical  organization  —  the  discussion  proceeds  in  logical 
steps  from  beginning  to  end,  without  irrelevancies  or  di¬ 
gressions.  One  way  to  insure  logical  organization  is  to 
start  with  an  outline.  Divide  the  correspondence  into  its 
major  parts  and  list  them  in  a  natural  order  (e.g.,  "State¬ 
ment  of  the  Problem",  "Alternate  Solutions",  "Recommended 
Course  of  Action" ) .  Then  list  the  various  things  to  be 
discussed  in  each  major  part,  as  if  you  were  designing  vu- 
graphs  to  brief  the  subject.  When  you  list  these  various 
things,  make  their  order  logical.  For  example,  if  you* re 
discussing  the  history  of  a  problem  or  a  test  program, 
discuss  it  the  way  it  happened  —  chronologically.  If 
you're  discussing  a  system's  capabilities,  discuss  them  in 
the  order  in  which  they  would  be  used  (e.g.,  detection, 
tracking,  engagement).  Once  you  have  established  an  order, 
stick  to  it.  For  example,  if  you  write  that  the  system  has 
parts  X,  Y,  and  Z,  don't  then  provide  detailed  descriptions 
of  Z,  X,  and  Y. 

b.  Wording .  A  logical  outline  provides  the  framework; 
words  expand  the  outline  into  a  completed  draft.  Do  not 
assume  that  just  any  old  words  will  do;  words  have  meaning 
and  deserve  to  be  selected  accordingly.  The  following  re¬ 
commendation  from  a  draft  Evaluation  Report  is  an  example  of 
near-random  use  of  words: 

"Stabilize  tracking  solutions,  lest  the  system  as 
non  functioning  will  not  be  militarily  useful  at  sea  in  the 
new  future  it  was  designed  for." 

This  extract  from  a  draft  comment  letter  is  another  example 
of  near-random  use  of  words: 

"Minimum  functional  performance  parameters  must  be 
specifically  stated  to  determine  operational  effectiveness 
acceptability. " 

Pay  attention  to  words  —  to  their  meanings.  Make  sentences 
say  what  you  mean. 


c.  Headings .  Writers  sometimes  ignore  the  meaning  of 
headings;  "Results"  paragraphs  end  up  full  of  conclusions;  a 


"Required  Operational  Characteristic"  is  presented  as  "Safety 
will  be  tested.  .  which  is  certainly  not  a  system  charac¬ 
teristic.  In  Evaluation  Reports,  the  tendency  to  ignore 
headings  increases  as  you  get  further  into  enclosure  (1). 

The  "Results  and  Discussion"  paragraphs  of  the  last  few  S- 
Tests  often  read  like  "Additional  Recommendations." 

d.  Adjectives  and  Adverbs.  Be  careful  of  these  things. 
They  tend  to  become  unjustified  superlatives,  or  they  inject 
bias  (or  the  appearance  of  bias).  Stick  to  nouns  or  verbs 
as  much  as  possible.  That  "The  system  is  degraded  by  .  .  ." 
may  be  obvious  from  test  results;  that  "The  system  is  seriously 
degraded  by.  .  ."  may  be  in  the  eye  of  the  beholder.  This 

is  not  to  say  that  COMOPTEVFOR  will  not  include  that  something 
is  "seriously  degraded."  He  may,  but  he  will  demand  that  it 
be  substantiated. 

e.  Acronyms .  Do  not  assume  that  by  defining  an  acronym 
you  have  told  the  reader  what  the  thing  is.  VAST  (Versatile 
Avionics  Shop  Test),  EXCAP  (expanded  capability),  and  ICAP 
(improved  capability)  have  very  little  information  content 
compared  to  MTBF  (mean  time  between  failures),  and  will 
probably  require  amplification  so  the  reader  understands 
what  is  being  discussed.  Too  many  acronyms  tend  to  make 
confusing,  boring  reading.  There  are  two  ways  you  can 
prevent  this: 

(1)  Use  plain  English  rather  than  trade  jargon. 

(2)  Never  use  an  acronym  if  it  will  appear  only 
once;  consider  not  using  acronyms  if  they  will  appear  only  a 
few  times,  particularly  if  their  appearances  will  be  widely 
separated  in  the  text. 

g.  Editing.  Editing  is  the  responsibility  of  the 
author,  and  it  is  the  responsibility  of  each  reviewer  of  a 
draft.  Failure  to  exercise  these  responsibilities  results  in 
proposals  like  the  following: 

"If  the  sensitivity  of  the  baseline  design  to  pro¬ 
vision  of  the  several  desired  performance  enhancements  is  to 
have  been  determined  during  the  concept  verification  phase, 
then  those  results  should  be  summarized  in  the  NDCP  for 
Milestone  I,  and  recommendations  made  as  to  which  should  be 
incorporated  into  the  validation  ba  eline  conceptual  de¬ 
sign.  " 

This  elegantly  phrased  nonsense  illustrates  the  most  im¬ 
portant  aspect  of  editing  —  making  sure  the  thing  makes 
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sense.  Putting  in  the  correct  hyphens  and  commas  is  important 
but  any  educated  person  can  do  that.  Making  sure  the  thing 
makes  sense  can  often  be  done  only  by  specialists  in  the 
subject  matter;  if  they  chop  nonsense  like  the  example, 
we're  in  trouble.  Read  aloud  what  you  have  written.  Have 
someone  else  read  it  aloud  to  you.  If  it  passes  both  those 
tests,  print  it! 

A102 .  General  Instructions  for  Correspondence 

a.  Before  a  document  is  released  for  review  it  should 
be  edited  ruthlessly  to  eliminate  nonessential  words  and 
phrases . 


b.  Underline  with  caution.  The  reader  should  be  in¬ 
telligent  enough  to  recognize  important  facts  without  having 
a  whole  paragraph  underlined  for  him.  Also,  underlines  may 
make  an  important  introductory  sentence  look  like  a  para¬ 
graph  heading.  Many  readers  ignore  headings. 

c.  Avoid  long  sentences.  Even  though  they  are  straight¬ 
forward  /~tEe~readerrmay~have  trouble  assimilating  their  con¬ 
tent.  Break  them  into  shorter  pieces  of  digestible  size, 
even  though  it  means  adding  a  word  or  two. 


d.  If  possible {  use  short  words  instead  of  long  words 
or  combinations  having  the  same  meaning. 


use 

therefore 

because 

estimate 


instead  of 
instead  of 
instead  of 
instead  of 


mation 


utilize 

for  this  reason 
due  to  the  fact 
make  an  approxi- 


e.  Use  words  in  common  use.  Don't  force  the  reader  to 
his  dictionary  by  using  an  uncommon  word  unless  no  other 
word  will  do. 


f.  Drafts  for  approval  should  be  double-spaced.  This 
allows  changes  and  corrections  to  be  made  neatly,  without 
awkward  writing  in  the  margins.  Neat,  readily  understood 
changes  can  go  to  the  Admiral  without  retyping. 

g.  Changes  and  corrections  of  moderate  length  should  be 
made  with  erasable  black  pencil,  not  with  pen  or  colored 
pencil. 


h.  Changes  of  more  than  moderate  length  (rewriting  more 
than  a  few  consecutive  lines)  should  be  made  by  having  the 
changed  version  typed  (double-spaced)  on  a  separate  piece  of 
paper,  which  should  then  be  scotch- taped  into  the  document. 

A103 .  Specific  Guidelines  for  Standardization  and  Accuracy 

a.  Use  of  Hyphens 

(1)  In  general,  use  a  hyphen  to  join  two  or  more 
words  serving  as  a  single  adjective  before  a  noun. 

110-volt  line  high-speed  turn 

30-foot  depth  long-distance  transmission 

signal-to-noise  ratio  computer-derived  data 

(2)  There  are  exceptions  to  this  general  rule: 

(a)  When  the  first  word  is  an  adverb  ending  in 
«ly«  which  obviously  is  going  to  operate  on  the  word  that 
follows  it: 

A  readily  available  spare 

An  increasingly  negative  reading 

(b)  In  commonly  used,  compound  adjectives,  now 
accepted  as  a  single  word:  (check  the  dictionary) 

lightweight  aircraft 

underwater  test 

(3)  Some  compound  nouns  (nouns  consisting  of  two  or 
more  words  that  name  one  subject)  are  hyphenated: 

light-year 

watt-hour 

Many  compound  nouns  omit  the  hyphen.  Many  have  been  fuzed 
into  single  words.  The  only  way  to  tell  is  to  check  the 
dictionary. 

(4)  When  it  is  necessary  to  spell  out  numbers  (for 
example,  the  first  word  of  a  sentence),  use  a  hyphen  for  the 
numbers  twenty-one  through  ninety-nine  (when  appropriate). 
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(5)  Do  not  use  hyphens  in  Mark/Mod  designations 
unless  you  are  quoting  a  reference  which  does: 

e . g . ,  Mk  4  Mod  3 

(6)  Hyphens  are  supposed  to  reduce  confusion  and 
ambiguity.  If  you  don't  like  them,  try  rephrasing  the 
sentence  to  eliminate  them: 

3 

"The  receiver  was  a  1-kg,  600-cm  computer-controlled 
system. " 

This  could  be  written: 

3 

"The  receiver  weighed  1  kg,  occupied  600  cm  ,  and 
was  controlled  by  a  computer." 

Note  that  the  first  sentence  is  both  more  direct  and  shorter. 

b.  Use  of  Symbols 

(1)  Use  "%"  instead  of  "percent."  (Except  on  messages.) 

(2)  Use  "°"  instead  of  "degrees." 

c.  Use  of  Numbers 


(1)  Watch  out  for  too  many  significant  figures.  Be 
sure  the  data  substantiate  the  significant  figures  you  use, 
and  that  they  are  really  necessary.  For  instance: 

"The  equipment  failed  to  demonstrate  the  required 
65-hour  MTBF.  The  actual  MTBF  was  4.38  hours,  with 
90%  confidence  that  it  is  at  least  1.67  hours." 

Test  data  may  substantiate  4.38  and  1.67  hours,  but  the 
point  is  that  "about  4  hours"  and  "about  2  hours"  are  no 
where  near  65  hours.  That's  what  we're  trying  to  get  across 
to  our  readers. 

(2)  Numbers  under  10  are  spelled  out  except  for  time 
and  measurement: 

"A  team  of  four  UDT  swimmers  completed  the  1-nmi 
course  in  1  hour  25  minutes." 

(3)  In  messages,  where  transmission  accuracy  is  not 
under  our  control,  critical  numbers  should  be  written  out. 
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(1)  COMOPTEVFOR  statements  concerning  aspects  of  a 
program  imply  our  having  already  considered  the  statement, 
and  the  word  "considered11  should  not  be  used.  For  example, 
instead  of  saying  "The  tests  are  considered  unsatisfactory 
for  the  following  reasons  . . . , "  word  the  statement  to  read 
"The  tests  are  unsatisfactory  ..." 

(2)  In  general,  the  word,  capability  is  used  for 
describing  machinery /hardware/software/things .  Ability  is 
generally  associated  with  people. 

(3)  Don't  say  "observe"  when  you  mean  "monitor."  We 
"monitor"  DA  testing  for  IOT&E  in  conjunction  with  DT&E.  We 
"observe"  test  scenarios,  firing  exercises,  aircraft  flights, 
equipment  operation,  etc. 

(4)  "Defensive  Blocking"  and  "Shouldering”.  These 
terms  have  been  used  to  describe  maneuvers  by  our  ships  to 
restrict  undesirable  Soviet  ship  positioning.  While  fully 
understood  by  Navy  personnel,  to  outsiders  the  terms  suggest 
a  form  of  physical  contact  which  can  be  provocative  or 
harassing.  In  order  to  preclude  such  interpretations,  the 
standard  naval  term  "screening"  is  used. 

(5)  "Hicrh-Value  Target"  and  "High-Value  Unit.” 

These  terms  define  the  U.S.  Navy's  principal  ships  from  an 
enemy's  point  of  view,  i.e.,  as  potential  targets.  The  CNO 
considers  it  important  that  we  emphasize  our  offensive 
capabilities,  as  opposed  to  our  defensive  requirements,  when 
we  refer  to  Navy  units.  This  can  be  accomplished,  in  part, 
through  the  use  of  specific  ship  designations  such  as  CV  or 
LPH;  in  cases  where  a  generic  term  is  required,  use  "main 
body"  in  place  of  "high-value  target"  or  "unit" . 


(6)  "Overflight. ”  While  this  term  may  be  applicable 
to  boundaries,  coastlines,  land  masses  and  other  fixed 
objects  or  large  reference  points,  it  is  not  appropriate  to 
use  it  with  reference  to  ships.  At  high  and  mid  altitudes, 
it  is  difficult  to  determine  if  an  aircraft  flew  precisely 
over  a  ship.  In  addition  to  the  questionable  accuracy  of 
the  term,  the  word  connotes  a  scene  where  a  helpless  ship  is 
subjected  to  an  unchallenged  aircraft.  Since,  in  most 
instances,  the  aircraft  is  probably  conducting  a  surveillance 
mission,  its  action  should  be  referred  to  as  having  conducted 
surveillance  of  the  subject  ship. 


e.  Use  of  Acronyms 


(1)  In  COMOPTEVFOR  correspondence  the  acronym  precedes 
the  definition;  e.g.,  MTBF  (mean  time  between  failures).  If 
the  definition  is  only  a  collection  of  words,  these  words 

are  written  in  lower  case.  If  the  definition  is  a  proper 
name  of  an  equipment  or  system,  use  upper  case;  e.g.,  VAST 
(Versatile  Avionics  Shop  Test).  Do  not  hyphenate  definitions 
for  MTBF,  MTTR,  etc. 

(2)  AS CM  (anti-ship-capable  missile)  is  not  to  be 
used  in  OPTEVFOR  correspondence. 

f.  Listing.  When  listing  objectives,  evaluation  criteria, 
etc.,  in  subparagraphs,  end  each  subparagraph  with  a  period, 
not  a  comma  or  semicolon.  For  example: 

"3.  The  DA  will  provide: 

a.  Certification  of  readiness  for  OPEVAL. 

b.  Technical  support  as  required. 


g.  Preparation  Dates  on  Outgoing  Correspondence 

(1)  Messages.  It  appears  in  the  lower  right-hand 
corner  (the  same  area  used  for  the  ’chop'  ladder). 

(2)  Rough  Correspondence.  It  appears  in  the  lower 
right-hand  comer. 

(3)  Smooth  Correspondence.  It  appears  on  the  green 
copy  sheet  and  is  the  same  as  the  date  of  ACOS  chop. 

h.  Message  Nit  Picks.  Whenever  possible  articles 
(e.g.,  a,  an,  the)  are  avoided  in  messages.  Use  of  short 
verbs  is  encouraged  (e.g.,  is,  were,  will).  Instead  of 
saying  "originator11  in  the  text  of  messages,  use.  "COMOPTEVFOR." 

i.  Use  of  Metric  Units 


(1)  The  Department  of  Defense  has  directed  use  of 
the  international  metric  system  in  all  activities  consistent 
with  operational,  economical,  technical,  and  safety  consider¬ 
ations.  Specifically  directed  was  that  technical  reports, 
studies,  and  position  papers  include  metric  units  of  measure¬ 
ment  in  addition  to  or  in  lieu  of  U.S.  customary  units. 


(2)  Planning  documents  (DCPs,  TEMPs,  Test  Plans, 

ass?ciated  with  0T&E  are  considered  to  be 
*n  technical  reports ,  studies ,  and  position  papers" 
and  will  use  international  metric  units  as  the  primary 

necessary?tS '  Wlth  U'S’  custoinary  units  in  parentheses  where 

E  380-7^i3^T?Q  S?D"aKPr°Y«J  metric  system  described  in  ASTM 
fhl  ?  November  1973  will  be  followed,  except  that 

the  traditional  Navy  units  "nmi"  and  "knot"  will  not  be 
transformed  to  metric  in  COMOPTEVFOR  documents.  - 


Annex  B 


Survivability /Vulnerability  Guidelines 

B101.  Introduction.  While  the  details  of  S/V  (survivabil-  (R 

ity/vulnerability)  evaluation  vary  considerably,  depending 
on  the  system,  its  function,  etc.,  the  fundamental  steps  an 
OTD  must  take  with  regard  to  S/V  are  essentially  constant. 

They  are: 

a.  Determine  if  S/V  is  an  issue  in  the  OR.  If  it  is 
not,  but  it  appears  that  it  should  be,  take  action  to  bring 
this  to  CNO's  attention. 

b.  Determine  that  if  a  threat  statement  is  required  by 
OPNAVINST  3811.1  series,  it  is  in  hand.  (For  assistance,  con¬ 
tact  the  Force  Intelligence  Officer  (Code  24).) 

c.  Ensure  that  the  NDCP  and  TEMP  address  S/V  in  sufficient 
detail;  that  necessary  S/V  criteria  are  provided  in  each  docu¬ 
ment;  and  that  the  S/V  criteria  provide  sufficient  guidance 
for  the  DA,  contractor,  and  OTD. 

d.  In  the  TEMP  and  related  DT&E  planning  documents, 
ensure  that  the  DA  is  committed  to  testing  that  will  support 
an  S/V  evaluation,  and  that  assets  to  demonstrate  criteria 
are  identified  (ranges,  targets,  etc.). 

e.  Ensure  that  OT&E  addresses  the  real  threat,  using 
realistic  targets  and  countermeasures. 

f.  The  bottom  line  is  this  —  determine  if  the  system 
will  complete  its  mission.  If  it  will,  it's  survivable  and 
not  overly  vulnerable  <.o  hostile  actions. 

B102.  S/V  References  and  Points-of-Contact  (A 

a.  The  following  is  an  incomplete  listing  of  S/V  ref¬ 
erence  documents.  For  help  in  using  them,  see  your  analyst 
or  02T. 


(1)  General  S/V  Guidance 

(a)  DOD  Directive  5000.3. 

(b)  OPNAVINST  3960.10. 

(c)  CNM  ltr  dated  20  Apr  1978  (repetitive  errors 
in  TEMPs).  (See  paragraph  B103.C.  below.) 
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(d)  OPNAVINST  3811.1  (threat  statements). 

(e)  NAVMATINST  3882.2  (threat  statements). 

(f)  NAVMATINST  3900.16  (NAVMAT  Combat  Survivability 


Program) . 

(2 )  Electromagnetic  S/V  Guidance 

(a)  DOD  Directive  C4600.3. 

(b)  SECNAVINST  C3430.2  (ECCM). 

(c)  CNM  ltr  ser  218  of  13  Mar  1979  (E3  (electro¬ 
magnetic  environmental  effects)  in  acquisition  programs). 

(d)  NAVMATINST  C3430.3  (ECCM). 

(e)  NAVMATNOTE  2400  of  5  Dec  1978  (E3  in  acquisi¬ 
tion  programs ) . 

(f)  CNO  ltr  987/P6/69884  of  25  Nov  1975. 

(g)  NAVMATINST  2410.2  (EMP). 

(h)  CNO  memo  987/P6/569885  of  1  Aug  1975  (EMP). 

(3)  Other  S/V  Guidance 

(a)  NAVMATINST  9110.2  (structural  firing  tests). 

(b)  NAVMATINST  9110.3  (shock  hardening  of  govern¬ 
ment-furnished  items). 

(c)  OPNAVINST  9010.300. 

(d)  NAVSURFWPNCENINST  8020.2  (explosives  used  in 
naval  weapons ) . 

(e)  SECNAVINST  S5430.86  (CBW). 

(f)  OPNAVINST  09110.2  (ship  shock  tests). 

(g)  NAVMATINST  09110.1  (ship  shock  tests). 

(h)  JTCG/AS-77-D-001  (aircraft  system  S/V  require¬ 
ments  ) . 

b.  S/V  points-of-contact  for  the  Navy  Labs  can  be  found 
in  Annex  C.  Within  the  Navy  Material  Command,  points-of-con- 
tact  are: 
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(1)  Dr  June  Amlie  (MAT-08D13):  AV  222-9015. 

(2)  Mr  Norm  Jackson  (MAT-08DE) :  AV  222-1887. 

c.  Liaison  with  the  Naval  Security  Group  (see  SEC- 
NAVINST  C3430.2)  should  be  initiated  by  letter.  (A  sample 
is  provided  at  the  rear  of  this  annex. )  The  point-of-con- 
tact  is  Mr.  Sam  Wong,  AV  292-0655. 

B103.  S/V  Aspects  of  Program  Documentation  (A 

a.  Review  OPNAVINST  3811.1. 

b.  The  directives  and  instructions  listed  in  paragraph 
B102  require  that  S/V  be  addressed  at  all  stages  of  develop¬ 
ment,  and  that  program  decuments  address  the  S/V  issues  that 
are  of  concern.  The  OTD  must  be  familiar  with  the  directions 
and  instructions  that  affect  his  system,  and  must  be  ready 

to  insist  —  early  on  —  that  their  provisions  be  adhered  to. 
Failure  to  do  so  may  well  result  in  an  inability  to  make  a 
meaningful  S/V  assessment. 

c.  The  Required  Technical  Characteristics  contained  in 

the  NDCP  and  TEMP  should  contain  a  complete  set  of  S/V  criteria. 
(Most  initial  drafts  are  devoid  of  S/V  criteria.)  CNM  ltr 
dated  20  Aug  1978  on  TEMP  errors  states,  "Make  sure  that  there 
is  a  list  of  key  technical/operational  characteristics,  showing 
performance  parameters,  goals,  and  thresholds.  The  primary 
errors  in  this  area  have  been  related  to  inadequate  R&M  (Re¬ 
liability  and  Maintainability)  requirements  and  lack  of  or 
inadequate  S/V  requirements.  Technical  characteristics 
should  include  S/V  requirements  and  design  R&M  requirements ..." 
Furthermore,  the  NDCP  is  required  to  present  the  various 
alternatives,  including  S/V  tradeoffs. 

d.  Part  III  of  the  TEMP  should  outline  the  DA's  plan  of 
action;  insist  that  it  specify  how  system  survivability  will 
be  demonstrated. 

e.  The  Required  Operational  Characteristics  should  be 
based  on  the  mission  of  the  system  and  the  combat  environment 
in  which  the  system  is  expected  to  operate.  (See  Codes  24, 

02T,  and  02B,  your  analyst,  and  fellow  OTDs  for  assistance.) 

B104.  The  E-Tests 


a.  Testing  for  S/V  should  be  approached  in  two  ways: 
that  which  can  actually  be  done  during  active  project  opera¬ 
tions,  and  that  which  cannot.  The  first  should  be  accoiqplished 
as  in  Test  E-3  in  the  sample  Test  Plan  of  Section  10. 
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b.  The  scenario-related  E-tests  must  be  designed  to 
demonstrate  the  system's  capability  —  or  lack  of  capability 
—  to  accomplish  its  mission  in  the  intended  environment. 

You  should  employ  the  best  tactics  (ours  and  theirs)  avail¬ 
able  and  the  best  countermeasures  (ours  and  theirs)  available. 
Targets  must  be  realistic,  and  must  be  employed  realistically. 
(Try  to  translate  BQM-34s  destroyed  one-at-a-time  on  a  sunny 
day,  with  flat  seas,  clear  Med  environment  in  June  to  destroy¬ 
ing  a  raid  of  four  AS-6s  with  jamming  in  the  North  Atlantic 

in  January.) 

c.  Test  E-3  in  the  sample  Test  Plan  focuses  on  the  "cheap 
kill"  aspect  of  system  survivability.  Here  is  where  a  majority 
of  the  DA's  S/V  test  results  can  be  useful;  they  will  be  avail¬ 
able  only  if  you  have  ensured  beforehand  that  the  DA  will  con¬ 
duct  the  required  tests.  (For  example,  using  the  model  shown 
in  Figure  B-l.)  Equipped  with  the  threat  statement,  required 
operational  characteristics,  knowledge  of  test  results  to  be 
expected  from  the  DA  and  NAVSECGRU,  and  the  list  of  considera¬ 
tions  of  paragraph  B105,  you  should  be  able  to  modify  the  sam¬ 
ple  Test  E-3  to  fit  your  needs.  You  should  concentrate  on 
issues  that  you  cannot  actually  test.  If  technical  questions 
arise,  consult  the  Navy  Lab  people  (see  Annex  C  or  the  HSAP 
representative) . 

B105 .  Survivability /Vulnerability  Considerations 
a.  Consider  degradation  from: 

(1)  Man-made  Environment 

(a)  Blast  effects  of  underwater,  air,  contact, 
or  penetrating  conventional  high-explosives  and  nuclear 
weapons . 

(b)  Fragment  damage  (primary  and  secondary) . 

(c)  Progressive  fire/ flooding/component  failure. 

(d)  Small  arms  fire  (primary  and  secondary 

effects) . 

(e)  Chemical  and  biological  weapons  effects. 

(f)  Nuclear  radiation  and  EMP  effects. 

(g)  Thermal  effects  (nuclear,  laser,  and  on¬ 
board  fire) . 

(h)  Interference  (enemy,  friendly,  or  intra¬ 
platform)  : 
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1.  Laser. 


2.  Electromagnetic  (EMI) . 

3_.  Chaff. 

£.  Smoke,  dust  (detonation  or  created) . 

(i)  Early  burst  of  own  weapons. 

(j)  Mistargeting  of  own  weapons. 

(k)  Sympathetic  detonations. 

(2)  Natural  Environments  (Note:  Equipment  performance 
in  the  natural  environment  is  evaluated  under  "Compatability. " 
The  effects  of  the  natural  environment  are  also  considered 
here,  because  they  can  enhance  or  diminish  the  simultaneous 
effects  of  enemy  action.) 

(a)  Sea  state  (wave  dynamics,  salt  spray,  green 
water,  shock,  and  vibration) . 

(b)  Weather  conditions  (rain,  hail,  clouds, 
haze,  fog,  dust,  and  wind) . 

(c)  Climate  (temperature  extremes,  temperature 
cycles,  and  humidity) . 

(3)  Enemy  Countertactics 

(a)  Detection  of  platform  or  weapon  system 
signatures  (visible,  infrared,  magnetic,  noise,  wake,  and 
electromagnetic) . 

(b)  Interference  (smoke,  dust,  camouflage). 

(c)  EMCON. 

b.  A  ship  (or  weapon  system)  is  survivable  if  it  is: 

(1)  Difficult  to  detect,  classify,  or  track.  The 
following  elements  pertain: 

(a)  Radar  cross  section  control. 

(b)  IR  signature  control. 

(c)  Visible  signature  control. 

(d)  UV  signature  control. 
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(e)  Electronic  signature  control  (EMCOM) . 
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(£)  Acoustic  signature  control. 

(g)  Radio  frequency  signature  control. 

(h)  Wake  control. 

(2)  Difficult  to  hit.  The  following  pertain: 

(a)  Size,  maneuverability,  and  speed. 

(b)  1R,  optical,  UV,  electronic,  and  active 
weapons  countermeasures. 

(c)  Torpedo,  missile,  and  gunfire  launch  warning. 

(3)  Difficult  to  damage.  The  following  pertain: 

(a)  Shock,  fire,  and  fragment  protection. 

(b)  Compartmentation. 

(c)  Stability  margins. 

(d)  CW/BW  protection. 

(e)  Air  blast  protection. 

(f)  Redundant  critical  components  and  paths. 

(g)  Reduced  mode  operation. 

(h)  Armor  and  internal  blast  protection. 

(i)  Small-caliber  projectile  protection. 

(j)  Fuzing  plates. 

(k)  Critical  personnel  redundancy,  and  personnel 

protection. 

(l)  Magazine  fire  protection. 

(m)  Survivable  communications  (internal  and 

external) . 

(n)  Laser  protection. 

(4)  Easy  to  repair.  The  following  pertain: 

(a)  Quick-fix  critical  cables. 

(b)  Seaworthiness  patches. 
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(c)  Personnel  survival. 

(d)  Critical  spares. 

(e)  Modular  construction. 

(£)  Alternate  lighting. 

(g)  Survivable  communications. 

B106 .  Assessing  a  Ship* a  Physical  Vulnerability* 

a.  Survey  topside  electronics  and  assess  their  vulner¬ 
ability  in  light  of  operational  experience  (e.g.,  Vietnam) 
and  associated  studies  (e.g.,  CG  32  and  FFG  7).  Recommend 
corrective  measures . 

b.  Review  results  of  the  EMP  test  of  the  ship  at  the 
EMPRESS  (Electromagnetic  Pulse  Radiation  Environment  Simu¬ 
lator  for  Ships)  facility  with  full  electronics  and  other 
mission-critical  equipment  in  operation.  Identify  deficiencies 
in  equipment  and  system  design,  recommend  corrective  action, 
and  develop  operational  procedures  for  minimizing  mission 
degradation  by  EMP. 

c.  Assess  the  adequacy  of  shock  resistance  of  ship 
systems  in  light  of  results  from  shock  trials,  and  evaluate 
corrective  measures  taken.  Analyze  results  of  shock  tests 
vis-a-vis  current  and  projected  enemy  weapons. 

d.  Identify  deficiencies  in  equipment  arrangements  and 
locations  that  may  compromise  the  ship's  survivability. 
Recommend  corrective  measures  (e.g.,  segregation  of  systems 
operating  in  parallel  and  consolidation  of  systems  operating 
in  series) . 

e.  Identify  critical  elements  (including  subsystem 
paths)  in  the  operational  sequence  whose  inactivation  re¬ 
sults  in  high-level  degradation  of  system  performance. 

Recommend  corrective  measures  (e.g.,  redundancy,  manual 
override) . 

f.  Identify  deficiences  in  damage  control  measures 
(e.g.,  fires,  flooding),  and  equipment  location  and  protection 


"keference:  b'TNSkbc  Technical  Report  76-0116,  Vulner¬ 
ability  Issues  in  Total  Ship  System  Opera¬ 
tional  Test  6  Evaluation,  Sept  1976  (See 
Figure  B-2) 
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in  light  of  operational  and  damage  experiences  (e.g.,  BELKNAP , 
FORRESTAL,  etc.).  Recommend  corrective  measures. 

g.  Conduct  at-sea  tests,  using  non-toxic  smoke  or  gas 
agents,  of  the  adequacy  of  personnel  protection  measures 
against  chemical,  biological,  and  radiation  weapon  effects. 
Recommend  corrective  measures. 

h.  Conduct  at-sea  operations  of  the  "damaged"  ship  to 
simulate  degraded  performance  states  for  different  types  and 
levels  of  impairment  that  may  result  from  weapon  effects. 
Assess  total  ship  operational  effectiveness  in  these  degraded 
modes . 

B107.  Assessing  a  Ship's  Signature 

a.  Conduct  an  EMPASS  (electromagnetic  performance  of 
aircraft  and  ship  systems) -type  survey  of  the  ship.  Develop 
tactics  to  minimize  vulnerability  because  of  intentional 
radiations  of  shipborne  equipment. 

b.  Conduct  radar  ranging  of  the  ship  at  the  RAM  (radar 
area  measurement)  site  in  the  Chesapeake  Bay  to  determine 
the  far-field  radar  cross-section  as  seen  by  enemy  search 
and  targeting  radars.  Conduct  a  near-field  radar  survey  to 
identify  critical  points/areas  (highlights)  of  the  ship  as 
seen  by  an  active  radar  homing-missile  seeker.  Propose 
modifications  to  eliminate/reduce  critical  highlights,  and 
tactics  to  minimize  vulnerability  to  active  radar  sensors 
and  weapon  systems. 

c.  Conduct  an  IR/EO  survey  of  the  ship  using  captive- 
missile-seeker  test  procedures.  Propose  tactics  to  minimize 
vulnerability. 

d.  Analyze  data  from  ship  noise  trials.  Evaluate  the 
impact  of  own-ship  noise  on  friendly  ship  sonars  and  sonobuoys 
Propose  preliminary  tactics  to  minimize  acoustic  vulnerability 

e.  Conduct  magnetic  and  pressure  profiling  of  the  ship 
and  assess  the  ship's  vulnerability  to  underwater  influence- 
fuzed  weapons  (e.g.,  mines). 

f.  Evaluate  the  effectiveness  of  suppression  techniques 
installed  on  the  ship  to  reduce  ship  emissions  and  radiations 
in  the  electromagnetic  and  acoustic  frequency  ranges. 
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DEPARTMENT  OF  THE  NAVY 

COMMANDER  OPERATIONAL  TEST  AND  EVALUATION  FORCE 
NORFOLK,  VIRGINIA  23511 


644:kmb 
3960 
Ser  XXX 
Date 


From:  Commander  Operational  Test  and  Evaluation  Force 
To:  Commander  Naval  Security  Group 

Sub j :  Signal  Susceptibility  and  Vulnerability  Assessment 
of  the  New  Weapon  System 

Ref:  (a)  SENAVINST  C3430.2 

1.  In  accordance  with  the  provisions  of  reference  (a) ,  it 
is  requested  that  the  Naval  Security  Group  conduct  a  signal 
susceptibility  and  signal  vulnerability  assessment  of  the 
New  Weapon  System  during  the  operational  evaluation  currently 
scheduled  23  September  -  20  December  19XX.  Point  of  contact 
for  resource  identification  and  coordination  is  LCDR  J.  A. 
MALO,  DEPCOMOPTEVFORPAC ,  Code  604A  (A/V  951-5531)  or  LCDR 
L.  W.  BAUER,  COMOP TEVFOR ,  Code  644  (A/V  690-5021) . 


ACOS 

By  direction 


Copy  to: 

(DA) 

CNO  (Sponsor) 

(OP-983) 

DEPCOMOPTEVFORP AX/VX-  ( OTD ) 
CHNAVMAT  (MAT-08D13) 
(MAT-08DE) 


SAMPLE  S/V  LETTER  TO  NAVSECGRU 


(Change  1) 
REVERSE,  BLANK 
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Annex  C 


Navy  S/V  Teat  Facilities 
C101.  Summary  of  Navy  Labs 

a.  NADC.  Naval  Air  Development  Center/  Warminster/  PA/ 

18974;  Info  Operator:  AV  441-2000?  NSAP  Coordinators  AV 
441-3100. 

Mission:  Principal  RDT&E  center  for  naval  aircraft 

systems  (less  aircraft-launched  weapons  systems) . 

Areas:  Failure  mode  effects  analysis/  SAM  and  AAA  survi¬ 

vability  modeling/  aircraft  combat  loss  analysis, 
aircraft  failure  analysis. 

b.  NSCC.  Naval  Coastal  Systems  Center,  Panama  City,  (R 

FL,  324017  Info  Operator:  AV  436-4011;  NSAP  Coordinator: 

AV  436-4204. 

Mission:  Principal  Navy  RDT&E  activity  in  support  of  naval 

missions  and  operations  that  take  place  primarily 
in  the  coastal  (continental  shelf)  regions. 

Includes  RDT&E  for  mine  countermeasures,  diving 
and  salvage;  coastal  amd^ihshpre  defense  (less 
ASW) ,  swimmer  operations,  and  amphibious  operations. 

Areas:  Mine  countermeasures  effects  analysis  (acoustic 

vulnerability  analysis) ,  torpedo  CM  effects 
analysis,  hostile  swimmer  CM  effects  analysis. 

c.  NOSC.  Naval  Ocean  Systems  Center,  San  Diego,  CA,  (R 

92152;  Info  Operator:  AV  933-1011?  NSAP  Coordinator:  AV 
933-2851. 

Mission:  Prime  Navy  RDT&E  center  for  C3  (command,  control 

and  communications);  ocean  surveillance?  surface- 
and-air-launched  undersea  weapon  systems;  and 
supporting  technologies .  - . 

Areas:  Systems  analysis,  torpedo  CM,  surveillance  sys¬ 

tems  (EM,  EO  signature  quantification) ,  EW  systems, 
SOSUS  improvement,  C3  architecture,  and  C3  land- 
based  test  site. 

d.  NPRDC.  Navy  Personnel  Reasearch  and  Development 
Center;  Info  Operator:  AV  933-1011;  NSAP  Coordinator:  AV 
933-7424. 
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Mission:  Prime  Navy  Activity  for  human  resource  RDT&E  in 

areas  of  manpower,  personnel,  education,  and 
training. 

Areas:  No  direct  survivability  work.  Human  factors  anal¬ 

ysis  possible,  however. 

e.  NSRDC .  Naval  Ship  Research  and  Development  Center 

(David  W.  fcaylor) ,  Bethesda,  MD,  20084;  Info  Operator:  AV 

287-0101;  NSAP  Coordinator:  AV  287-1681. 

Mission:  Prime  Navy  RDT&E  center  for  Navy  vehicles  and 

logistics. 

Areas:  Design  of  surface  combatants,  aircraft  carriers, 

craft,  and  submarines  against  effects  of  con¬ 
ventional  and  nuclear  weapons  delivered  under 
and  above  water.  Analysis  of  vulnerability  to 
these  forms  of  attack,  along  with  fire  and  own 
ship  ordnance  hazards.  Development  of  protection 
systems  against  external  and  internal  blast,  frag¬ 
ments,  and  underwater  shock.  These  include  air¬ 
craft  carrier  magazine  armor;  lightweight  armor 
systems  for  protecting  topsides  of  surface  combatants; 
spot  armoring  and  blast-resistant  design  of  search/ 
surveillance  antennas  and  weapons  system  directors; 
and  hull,  propulsion,  and  auxiliary  system  shock 
hardening.  Cost-benefit  analysis  of  protection 
options  and  survivability  features  such  as  arrange¬ 
ments  and  redundancy  are  also  included,  as  well  as 
characterization  of  threat  weapons  effects  and 
determination  of  ocean  environment  and  seakeeping 
characteristics  in  various  seaways  for  use  in 
combat  capability  assessments.  Surface  ship  and 
submarine  signature  control  efforts  include  surface 
ship  and  submarine  silencing  along  with  IR/EO/RCS 
and  magnetic  signature  control  as  well  as  determin¬ 
ing  antiship  IR  seeker  aimpoint  tracks.  Also 
military  effectiveness  studies  of  the  above  features. 

f.  NSWC.  Naval  Surface  Weapons  Center,  Dahlgren,  VA, 

22408;  Info  Operator:  AV  249-1110;  NSAP  Coordinator:  AV 

249-7164. 

Mission:  Prime  Navy  RDT&E  center  for  surface  ship  weapons 

systems,  ordnance,  mines,  and  strategic  systems 
support. 

Areas:  Platform  survivability  analysis  (threat  inter¬ 

actions)  ;  integrated  weapon  systems  S/V  analysis; 
weapons  effects  RDT&E;  EM  vulnerability  T&E; 
armor  and  materials  RDT&E;  CM/BW  S/V;  nuclear  wea- 
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pons  effects;  launchers,  guns,  electronics,  and 
electrical  subsystem  vulnerability;  ship  S/V  test 
and  analysis,  missile  S/V  test  and  analysis,  air¬ 
craft  S/V  test  and  analysis;  IR  and  EO  decoys; 

'  RF  decoys;  fuze  countermeasure. 

g.  NUSC.  Naval  Undersea  Systems  Center,  New  London,  CT, 

06320;  Info"" Operator :  AV  636-0111;  NSAP  Coordinators  AV 

636-2250. 

Mission:  Prime  Navy  RDT&E  center  for  submarine  warfare 

and  submarine  weapons  systems. 

Areas:  Torpedo  acoustic  noise,  C3  effectiveness,  ocean 

environment  (effects  on  detectability) ,  target 
strength,  force  S/V  analysis,  threat  detection, 
communication  effects  on  S/V. 

h.  NWC.  Naval  Weapons  Center,  China  Lake,  CA,  93555; 

Info  Operator:  AV  245-9011;  NSAP  Coordinator:  AV  245-3793. 


Mission:  Prime  Navy  RDT&E  center  for  air  warfare  systems 

(except  ASW)  and  missile  weapons  systems. 

Areas:  EW,  ECM,  IRCM,  ARM,  ASMD,  c2,  target  detection, 

classification,  and  hardkill  systems  analysis 
and  development;  target  vulnerability  for  weapons 
effectiveness  (surface  tgts) ;  aircraft  survivabil- 
f  ity  T&E  (tactics,  threats,  computer  description, 

vulnerability  assessment,  S/V  technology) . 


C102 .  Capabilities  and  Facilities  for  Measuring 
natures 


Siq- 


a.  Measurements  of  ship  (and  submarine)  radiated-noise 
signatures  in  acoustic  frequency  bands  for  sonar,  torpedo, 
and  mine  threat  assessments  are  conducted  on  several  calli- 
brated  ranges,  either  Navy-owned  or  available  to  the  Navy. 


(1)  East  Coast 


(a)  AUTEC  (Atlantic  Underwater  Test  and  Evalu¬ 
ation  Center),  Andros  Island  in  the  Bahamas,  located  off  the 
Florida  east  coast,  is  a  Navy-owned  facility  under  the 
command  of  NUSC/New  London.  The  AUTEC  acoustic  range  is  a 
deep-water  (>  1000  ft)  facility  using  a  fixed  hydrophone 
array  moored  off  the  bottom.  NUSC  is  responsible  for  sched¬ 
uling  use  of  the  range  and  for  providing  all  support. 


(b)  MONOB  (Mobile  Noise  Barge) ,  located  off  Port 
Everglades,  Florida,  is  a  Navy-owned  facility  designed  and 
dedicated  to  ship  (and  submarine)  acoustic  trials  by  and  for 
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NSRDC.  Other  users  can  have  access  to  MONOB,  but  on  a  not- 
to-interfere  basis.  MONOB  can  be  moved  around;  acoustic 
measurements  for  submarines,  for  example,  are  normally  made 
from  a  deep-water  moor  either  in  Exuma  Sound  or  in  the 
TOTO  (Tongue-of-the-Ocean) ,  both  of  6,000-ft  depth.  This 
facility  is  under  the  command  of  NSRDC/Car derock. 

(2 )  West  Coast 


(a)  SCARF  (Santa  Cruz  Acoustic  Range  Facility), 
located  at  Santa  Cruz  Island  20  miles  off  the  California 
coast,  is  a  private  facility  owned  and  managed  by  tJie  General 
Motors  Corp.  All  activities  including  data  acquisition  are 
carried  out  by  GM  under  contact  with  Navy  and  other  (industry) 
users.  The  SCARF  uses  a  deep-water  (>  1000  ft)  moored 

fixed  hydrophone  array.  The  Puget  Sound  Naval  Shipyard 
is  the  Navy's  principal  user  of  SCARF. 

(b)  Carr  Inlet,  located  off  Fox  Island  20  miles 
from  Bremerton,  Washington,  is  a  Navy-owned  facility  under 
the  command  of  PSNSY  (Puget  Sound  Naval  Shipyard).  It  is  a 
relatively  shallow-water  (<  390  ft)  facility  that  employs 

a  moored  fixed  hydrophone  array  system.  Normally,  the 
hydrophone  elements  are  located  at  depths  of  50,  100,  200, 
and  300  ft.  SSRNM  (Surface  Ship  Radiation  Noise  Measure¬ 
ment)  operational  test  facilities  at  San  Clemente,  California, 
supplement  the  BARSTUR  and  GARF  facilities  discussed  below. 

(c)  BARSTUR  (Barking  Sands  Tactical  Underwater 
Range ) ,  located  off  the  island  of  Kauai ,  Hawaii ,  is  a  Navy- 
owned  facility  primarily  used  as  a  weapons  tracking  range 
and  for  conducting  fleet  exercises.  It  is  a  calibrated 
range  with  a  capability  for  precise  tracking  of  missiles  (by 
surface  radars)  and  torpedoes  (by  underwater  fixed  hydro¬ 
phones),  and  for  precise  position  fixing  of  ships  and  sub¬ 
marines.  A  ship/sub  noise  measurement  capability  has  been 
added  to  BARSTUR.  This  range  comes  under  the  command  of 
PMTC . 


(d)  GARF  (Guam  Acoustic  Range  Facility)  is  a 
ship/submarine  noise  measurement  facility  off  the  island 
of  Guam.  It  is  under  the  command  of  LOGPAC. 

AUTEC,  MONOB,  SCARF,  Carr  Inlet,  and  GARF  acoustic  facilities 
are  calibrated  Navy-certified  ranges  and,  in  general,  all  use 
compatible  data  acquisition,  processing,  and  analysis  techniques 
for  ship  (and  submarine)  radiated-noise  measurements. 

b.  In-situ  acoustic  measurements  of  ship  radiated-noise 
using  line  hydrophone  arrays  and/or  sonobuoys  deployed  from 
a  work  boat  can  also  be  made  if  a  calibrated  range  cannot 


(Change  1) 


C-4 


be  used.  Such  measurements/  however,  are  limited  by  instru¬ 
mentation  inadequacies,  and  they  lack  the  precision  obtained 
when  using  a  calibrated  range  for  exact  position  (and  range) 
fixing.  In-situ  measurements  can  provide  useful  information 
with  regard  to  gross  acoustic  characteristics  and  deficien¬ 
cies  of  ships  such  as:  critical  tonals,  propellor  singing 
and  cavitation  inception  speeds,  etc. 

c.  The  capabilities  and  facilities  on  each  coast  for 
ship  IR/EO  signature  measurements  and  analysis  are  described 
below. 


(1)  East  Coast 


(a)  There  is  no  calibrated  range  for  making  IR 
signature  surveys  of  ships,  so  in-situ  techniques  have  to  be 
employed.  One  possibility  is  to  mount  a  captive  missile- 
seeker  in  a  helo  or  aircraft  and  fly  against  the  ship,  sim¬ 
ulating  a  missile  trajectory;  the  information  thus  collected 
pertains  only  to  the  particular  seeker  used.  Measurements 
of  a  broader  nature  need  to  be  made  using  wide-band  radio¬ 
meters  such  as  spectrometers/interferometers  to  get  the  spectral 
distribution  of  a  ship's  IR  signature,  and  imagers  to  determine 
the  spatial  distribution.  Another  possibility  is  to  instru¬ 
ment  the  EMPASS  aircraft  for  simultaneous  measurement  of  EM 

and  IR  signatures.  However,  it  would  be  more  appropriate  to 
use  a  helo  that  can  fly  at  low  altitudes  corresponding  to 
actual  missile  trajectories.  These  types  of  IR  measurement 
devices  could  be  located  on  some  fixed  land  station  with  a 
capability  for  "exact"  position  fixing  of  the  ship  target. 

NRL  is  currently  performing  IR  measurements  with  high  per¬ 
formance  aircraft  utilizing  dual-band  captive  seekers  and 
imagers  to  support  electronic  warfare  programs.  DTNSRDC  has 
made  IR  measurements  utilizing  thermocouples  on  a  ship  before 
it  conducted  passes  through  the  NRL  RAM  site  located  at  their 
Chesapeake  Bay  Division,  Maryland.  DTNSRDC  and  NRL  possess 
the  expertise  and  experience  to  assist  OPTEVFOR  in  planning  and 
carrying  out  ship  IR  measurement  programs. 

(b)  The  NRL  mobile  radiation  laboratory  is  routinely 
used  to  obtain  high  spectral  resolution  (0^5  cm”x)  EO  signature 
data.  Other  ongoing  activities  in  support  of  signature  measure¬ 
ments  utilize  unique  NRL  facilities  in  the  areas  of  E) /meteor¬ 
ology  and  atmospheric  optical  properties. 

(c)  The  joint  NRL  Electro-Optical  Technology  Pro¬ 
gram  Office/MIT  (Lincoln  Laboratory)  ship  IR  measurement  pro¬ 
gram  has  produced  a  data  base  of  high  spatial  resolution, 
radiometric,  IR  ship  signatures  in  both  important  atmospheric 
IR  spectral  windows  (3-5  ym  and  8-12  ym) .  Calibrated  imagery 
of  many  modern  Navy  ships  is  being  used  to>  study  geometrical, 
meteorological,  diurnal,  and  other  parameters  effecting  IR 
emission  from  ships,  as  well  as  IR  signattee  computer  models, 
and  sensor  ddsign. 


C-5 


(2)  West  Coast.  The  same  general  statements  with 
regard  to  the  East  Coast  also  apply  here. 

d.  Radar  Reflectivity 

(1)  East  Coast.  Ship  radar  reflectivity  character¬ 
istics  are  highly  aspect-dependent  in  terms  of  spatial  and 
spectral  distributions.  They  are  also  a  function  of  polari¬ 
zation,  elevation  angle  (as  might  be  seen  by  incoming  mis¬ 
siles  on  different  trajectories) ,  and  frequency.  The  Navy- 
owned  facility  for  making  measurements  of  ship  RCS  (radar 
cross-section)  from  a  fixed  ground  location  is  the  RAM  site 
located  on  the  Chesapeake  Bay,  Maryland.  This  is  a  calibrated 
range  possessing  the  necessary  position-fixing  capabilities. 

It  is  under  the  command  of  NRL.  Only  RCS  measurements  of 
ships  (as  might  be  seen  by  a  surveillance/detection  radar 
sensor)  are  made  at  the  RAM  site.  With  regard  to  what  ship 
radar  reflectivity  measurements  could/ should  be  made,  and  by 
whom,  the  following  is  offered: 

(a)  Aspect-dependent  (around  360*)  RCS  fine- 
grain  measurements  within  potential  threat  radar  bands, 
using  the  RAM  site  should  be  considered;  measurements  in  the 
L,  S,  C,  and  X-bands  could  be  accomplished.  East  Coast 
ships  could  be  scheduled  to  make  passes  through  this  measurement 
range . 


(b)  The  RAM  site  permits  ship  RCS  measurements 
at  near -grazing  angles  (to  the  horizontal) .  To  make  RCS 
measurements  for  elevation  angles  up  to  and  exceeding  45°, 
it  is  necessary  to  use  an  instrumented  airborne  platform 
(e.g.,  EMPASS  aircraft).  Ship  RCS  measurements  patterned 
after  the  RAM  ground  system  type  of  data  could  be  made  using 
an  airplane  platform  —  range  measurements  could  be  stored 
on  magnetic  tape. 

(c)  Another  ship  radar  reflectivity  character¬ 
istic  is  its  doppler  signature.  NRL  has  an  instrumented  X- 
band  system,  designed  primarily  for  recording  doppler  signa¬ 
tures  of  aircraft,  that  could  be  adopted  for  ship  measure¬ 
ments.  Doppler  characteristics  of  a  ship  may  be  thought  of 
as  second-order  vis-a-vis  RCS,  but  they  can  be  important 
from  the  point  of  view  of  classification  clues., 

(2)  West  Coast.  NELC  possesses  a  capability  for 
making  ship  R<SS'  far- fie  Id  measurements  in  the  X-band.  They 
also  possess  an  experimental  S-band  radar  system  which  could 
be  used  for  ship  RCS  measurements.  The  NELC  installations 
are  situated  along  the  California  coast.  The  radars  are 
located  100  ft  above  ground  level  and  permit  only  near¬ 
grazing  angle  measurements. 
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e.  With  regard  to  ship  magnetic  signatures ,  the  Navy 
possesses  several  degaussing  and  depending  ranges  on  the 
east  and  west  coasts  that  are  used  for  making  signature 
'.measurements.  STNSRDC  is  the  Navy's  lead  laboratory  for 
submarine  magnetic  silencing,  NSWC  for  surface  ships. 

There  is  no  counterpart  laboratory  on  the  West  Coast. 

Other  Navy  laboratories  participating  in  these  programs 
include  NUSC,  NADC  and  NCSC.  Also,  NSWC  has  specialized 
interests  in  ship  (and  submarine)  magnetics  from  the  point 
of  view  of  magnetic  mine  design.  A  description  of  the 
specific  capabilities  follows: 

(1)  East  Coast 

(a)  Two  degaussing  ranges  are  located  off  Norfolk  (R 
and  Charleston.  A  third  range,  primarily  for  destroyers, 
is  being  established  off  Mayport.  In  addition  to  these 
degaussing  ranges  for  transiting  ships,  the  Norfolk  facility 
has  arrays  of  bottom-mounted  magnetometers  for  making  mag¬ 
netic  measurements  of  moored  ships,  such  as  during  depending 
operations.  Charleston  has  specialized  facilities  for  measur¬ 
ing  and  calibrating  minesweepers.  Operation  of  the  degaussing 
ranges  comes  under  the  cognizance  of  the  Commanding  Officer 
of  the  Naval  Station  where  the  respective  ranges  are  located. 
Technical  guidance  on  the  kinds  of  information  and  procedures 
to  be  used  is  provided  by  NAVSEA.  DTNSRDC  provides  engineer¬ 
ing  support  for  hardware  and  software  improvements. 


(b)  NSWC  has  a  facility  off  Ft.  Lauderdale 
^nffor  acquiring  magnetic  signatures  of  ships  (and  submarines) . 


(c)  DTNSRDC  has  a  land-based  test  site  at  its  (R 

Annapolis  Laboratory  for  making  magnetic  measurements  on  full- 
sized  ship  equipment.  Electrical  motors  and  generators  can 
be  operated  under  typical  shipboard  supply  voltages  and  loads. 
Items  weighing  up  to  40  tons  can  be  investigated  with  arrays 
of  magnetometers  providing  171  magnetic  measurement  points. 


(2)  West  Coast.  The  two  degaussing  ranges  are  located  (r 
off  San  Diego  and  at  Pearl  Harbor.  The  San  Diego  facility 
has  the  additional  capability  of  arrays  of  bottom-mounted  mag¬ 
netometers  for  making  magnetic  measurements  of  moored  ships, 
both  steel-hulled  ships  and  minesweepers.  Operation,  admin¬ 
istrative  control,  technical  guidance,  and  range  improvements 
are  exercised  the  same  as  the  East  Coast  facilities. 


f.  Ship  Pressure  Signature  Measurements.  The  only  Navy 
laboratory  that  does  this  is  NSWc,  White  Oak.  They  possess 
a  semi-portable  instrumentation  package  that  has  been 
installed  in  Ft.  Lauderdale  and  Ft.  Monroe  test  facilities  - 
to  make  measurements  of  ship  pressure  signatures,  primarily 
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for  purposes  of  pressure-influence  mine  design.  The  NSWC 
instrumentation  package  can  be  installed  in  any  facility 
having  a  target  tracking  capability  such  as  exists  in  many 
of  the  ranges  discussed  in  previous  paragraphs;  or  it  can  be 
installed  in-situ,  so  long  as  some  kind  of  tracking  capa¬ 
bility  is  available  (it  is  critical  to  know  target  position, 
course,  aspect,  speed,  etc.  relative  to  the  array  of 
pressure-sensing  elements) .  NSWC  is  developing  a  self- 
contained  multi-influence  recording  system  (AUTO-r&nge) 
that  can  turn  itself  on-and-off  automatically  simply  by  the 
presence  of  a  ship  and  record  various  signatures  of  interest 
(acoustic,  magnetic,  and  pressure),  primarily  in  the 
influence-mine  frequency  range.  Generally  speaking,  "tar¬ 
gets  of  opportunity"  are  ranged  and,  more  seldom,  scheduled 
ship  runs  are  made  through  the  Ft.  Lauderdale  and  Ft.  Monroe 
facilities  to  acquire  pressure  signatures.  NSWC  possesses  a 
library  of  ship  pressure  signatures  from  dedicated  ranging 
runs  in  which  the  ship  motions  were  controlled.  The  Ft. 
Monroe  range  has  been  used  to  make  simultaneous  measurements 
of  ship  acoustic,  magnetic,  and  pressure  signatures. 

C103.  Navy  EMP  Simulation  Facilities 

a.  EMP  effects  are  one  of  the  more  important  nuclear 
threats  to  the  fleet.  EMP  from  a  high-altitude  detonation 
hundreds  or  even  thousands  of  miles  away  can  cause  permanent 
damage  or  temporary  operational  impairment  of  shipboard 
electronics  and  avionics  equipment.  EMP  technology  has  been 
steadily  developing  at  NSWC  since  it  became  the  Navy's  lead 
for  NWE  (nuclear  weapons  effects)  in  1969.  Because  of  the 
complexity  of  the  problem,  one  aspect  of  the  EMP  program  has 
been  development  of  EMP  simulation  facilities.  The  princi¬ 
pal  milestones  in  this  area  are: 

(1)  Established  the  EMPRESS  (Electromagnetic  Pulse 
Radiation  Environment  Simulator  for  Ships)  at  the  NSWC 
Solomons  Facility  in  1972. 

(2)  Established  the  EMPSAC  (EMP  Simulator  for  Air¬ 
craft)  at  NATC  in  1976.  Added  the  vertically  polarized 
NAVES  (Navy  Aircraft  Vulnerability  EMP  Simulator)  in  1977. 

b.  EMPRESS .  EMPRESS  is  a  subthreat-level  simulator 
designed  for  performing  coupling  studies  of  electrical/ 
electronic  systems  aboard  ships.  The  facility  is  capable 
of  illuminating  a  ship,  producing  EMP  polarized  either 
vertically  or  horizontally  and  including  both  high-;  ^ 
and  low-frequency  components  of  interest.  Although  the 
facility  is  primarily  for  subthreat-level  coupling  studies 
of  ships,  it  can  be  used  for  aircraft  fly-by  tests  and  near¬ 
threat  level  testing  of  small  subsystems  by  locating  the 
subsystems  close  to  the  facility  (i.e.,  within  50  meters). 
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d.  EMPSAC.  The  EMPSAC  facility  at  NATC  provides  a 
means  for  eMp  testing  of  naval  aircraft,  missiles,  and 
avionics  systems.  The  facility  uses  a  2.5  -  megavolt 
pulser  to  excite  antennas  that  provide  either  a  horizontally 
or  vertically  polarized  transient  EH  wave,  generally  at 
subthreat  levels . 

e.  NAVES.  NAVES  was  erected  at  NATC  in  1977  and  is  a 
vertically  polarized  conical  monopole  over  a  ground  plane. 

f.  Larger  Ships.  The  only  range  facilities  for  con¬ 
ducting  eMp  platform  hardening  effectiveness  tests  is 
EMPRESS.  The  EMPRESS  range  is  generally  not  adequate  for 
tests  of  large  ships  because  of  its  location;  therefore,  it 

is  planned  to  relocate  EMPRESS.  It  has  not  yet  been  established 
whether  the  EMPRESS  Facility  will  be  relocated  in  the  Atlantic 
or  in  the  Pacific. 
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Annex  D 


Intelligence  Support 

D101.  General .  OPNAVINST  3811. 1A  states  in  part  that  "for 
each  weapons  systems  program,  specific  planning  will  be 
included  for  obtaining,  updating  and  utilizing  threat 
support  throughout  the  life  cycle  of  the  program."  To 
assist  OTDs  in  obtaining  adequate  threat  support  for  OT&E, 
Headquarters,  DEPCOMOPTEVFORPAC,  VX-1,  VX-4,  VX5,  and  OPTEV- 
FORDET  Sunnyvale  have  either  full-time  Intelligence  Specialists 
(1630)  or  Collateral  Duty  Intelligence  Officers  assigned. 

( OPTEVFORDET  Hew  London  is  supported  by  Headquarters  and 
informally  by  intelligence  activities  in  the  Hew  London 
area ) . 

D102 .  Types  of  Intelligence  Available.  OTDs  should  be 
aware  of  two  types  of  intelligence:  Finished  Intelligence 
and  Operational  Intelligence. 

a.  Finished  Intelligence,  which  includes  published, 
agreed-upon  information  about  current  and  projected  enemy 
tactics  and  capabilities,  is  furnished  through  the  HWP-12 
series,  and  a  special  series  of  publications  produced  by  the 
Haval  Intelligence  Support  Center  expressly  for  the  RDT&E 
community  (see  paragraph  D105).  Additionally,  tailored 
threat  assessments  and  threat  support  plans  exist  which 
specifically  support  a  single  program  or  system.  The  HISC 
publications  and  tailored  threat  assessments  are  important 
because  they  represent  the  official  Havy  position  and  address 
the  projected  threat.  Since  the  majority  of  operational 
testing  involves  future  systems,  understanding  and  incorporating 
projected  threat  information  into  the  OT&E  process  is  imperative. 
Consequently,  frequent  use  of  Finished  Intelligence  publications 
by  OTDs  is  encouraged.  The  HWP-12  series  is  particularly 
important  for  test  scenario  development,  since  it  provides 
insight  into  enemy  tactics. 

b.  Operational  Intelligence,  which  deals  with  infor¬ 
mation  such  as  ship  positions,  satellite  surveillance 
periods,  and  location  of  foreign  intelligence  collectors,  is 
sent  by  message  upon  request  and  is  particularly  useful 
during  at-sea/range  testing  when  OPSEC  is  a  prime  concern. 

D103.  When  to  Use  Intelligence.  There  are  three  periods 
when  threat  information  is  particularly  important.  The 
first  is  during  the  TEMP  planning  stage.  Familiarity  with 
the  threat  at  this  point  will  help  an  OTD  anticipate  required 
OT&E  resources  (e.g.,  targets  and/or  simulators)  and  identify 


critical  T&E  issues.  The  second  important  period  is  during 
development  of  the  Test  Plan.  Updated  threat  information 
will  help  an  OTD  refine  requests  for  test  resouces  and 
develop  realistic  test  scenarios.  In  addition,  it  will 
provide  a  general  framework  upon  which  to  evaluate  the 
system.  Finally,  intelligence  support  should  be  considered 
during  at-sea/range  testing,  to  minimize  the  possibility  of 
compromise. 

D104.  OTD  Responsibilities.  The  Program  Sponsor  and 
CHNAVMAT  are  responsible  for  developing  an  intelligence 
support  plan  for  each  new  program.  However,  it  is  incumbent 
upon  every  OTD  to  ensure  that  a  threat  support  plan  exists 
and  that  essential  information  is  used  during  OT&E.  Frequent 
liaison  with  the  Intelligence  Officer  is  encouraged  and  is 
required  during  TEMP  and  Test  Plan  preparation.  If  it  is 
determined  that  existing  threat  support  is  inadequate, 
OPTEVFOR  intelligence  officers  will  assist  OTDs  in  obtaining 
needed  information. 

D105.  OPTEVFOR  intelligence  officers  will: 

a.  Forward  threat  support  requests  to  COMNAVINTCOM  in 
accordance  with  OPNAVINST  3811. 1A.  (Intelligence  officers 
at  subordinate  commands  will  forward  threat  support  requests 
to  COMOPTEVFOR  for  initial  coordination. ) 

b.  Ensure  that  Program  Sponsors,  Program  Coordinators, 
and  Program  Managers  receive  copies  of  all  OPTEVFOR  threat 
support  requests,  as  well  as  the  data  received  in  response 
to  those  requests. 

c.  Ensure  that  all  project-related  threat  data  are 
provided  to  the  OTD. 

d.  Provide  a  quarterly  report  to  COMOPTEVFOR  (ATTN: 

Code  24)  listing  the  threat  support  provided  and/or  planned 
for  each  assigned  project. 

e.  Provide  inputs  and  guidance  to  OTDs  on  threat 
matters  during  drafting  of  DCPs,  NDCPs,  TEMPs,  Test  Plans, 
and  T&E  reports. 

f.  Support  model  managers  by  coordinating  intelligence 
inputs  for  NWPs. 

g.  Support  the  production  of  OTGs  (OPTEVFOR  Tactics 
Guides)  and  tactical  memoranda  with  appropriate  intelligence 
information. 
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D106.  NISC  publications  recommended  for  use  by  OTDs: 

a.  Soviet  Naval  Threat  Circa  2000  -  DST-1200F-597-77. 

b.  Soviet  Threat  to  Air  Forces  -  DST-1300F-604-77 . 

c.  Soviet  Threat  to  Undersea  Forces  -  DST-1200F-598-77. 

d.  Soviet  Threat  to  Surface  Forces  -  DST-1200F-590-77. 

e.  Soviet  Ocean  Surveillance  Capabilities  -  DST-1430S- 
607-77. 

f.  Soviet  C3  Capabilities  -  DST-1270S-190-77. 

g.  Soviet  Air  Capabilities  -  DST-1300F-605-77. 

h.  Soviet  Submarine  Capabilities  -  DST-1220S-603-77. 

i.  Soviet  Surface  Capabilities  -  DST-1210S-602-77. 

j .  Soviet  AAW/ASMD  Capabilities  -  DST-1200S-601-77 . 

k.  Soviet  ASCM  Capabilities  -  DST-1330S-606-77 . 
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Guidelines  for  Assessing  Training  Plains 

E101.  Introduction.  NTPs  (Navy  Training  Plans)  are  prepared 
in  accordance  with  OPNAVINST  1500. 8H.  In  comment  letters  on 
"for  comment"  drafts  and  in  OPEVAL  reports,  OPTEVFOR  addresses 
the  adequacy  of  the  NTP . 

E102 .  Guidelines .  The  following  guidelines  are  provided 
for  reviewing  NTPs: 

a.  Operation  and  Maintenance  Training  Requirements 

(1)  The  first  step  is  to  verify  the  manning  required 
for  the  new  equipment.  The  DA  should  have  established 
manning  requirements  based  on  OPNAV  10P23  and  OPNAVINST 
1000. 16D.  The  operational  concept  for  the  new  equipment 
and,  therefore,  the  watch  station  requirements  at  Condition 
of  Readiness  I  and  III  will  impact  manning,  as  will  preventive 
and  corrective  maintenance  requirements.  Review  these 
requirements  with  the  DA. 

(2)  The  second  step  is  to  check  the  installation 
schedule.  The  installation  schedule  back-dated  by  the  required 
school  length  determines  the  required  training  schedule. 

(3)  Then  check  to  make  sure  the  required  training 
schedule  is  preceded  by  adequate  instructor  training. 

b.  Installation  Training  Requirements.  Make  sure 
installation  training  (if  required  for  the  new  equipment)  is 
provided  prior  to  delivery  at  the  installation  sites. 

c.  IMA  ( Intermediate  Maintenance  Activity}  and  Depot 
Training.  Make  sure  provision  is  made  for  training  IMA  and 
depot  repair  personnel,  consistent  with  the  maintenance 
concept  of  the  ILSP. 

d.  Training  Facilities 

(1)  Equipment  Delivery .  Ensure  the  delivery  schedule 
accommodates  the  requirements  for  operation  and  maintenance 
training. 

(2)  GPETE  (General  Purpose  Electronics  Test  Equipment) 
SPETE  (Special" Purpose  Electronics  Test  Equipment),  and  PSE 
(Pecular  Support  Equipment).  EnSlxe  that  training  site 
allowances  of  GPETE,  SPETE,  and  PSE  are  consistent  with 
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anticipated  shipboard  allowances.  Additionally,  ensure  that 
allowances  provide  for  continuous  availability  through  the 
calibration  cycle  of  training  equipment.  Ensure  that  cali¬ 
bration  provisions  have  been  made  for  equipment  using  new 
technology. 

e.  Training  Adequacy.  OT&E  reports  include  an  assess¬ 
ment  of  the  adequacy  of  planned  training.  This  evaluation 
may  be  limited  by  the  training  received  (i.e.,  not  as  specified 
in  the  NTP),  but  the  training  that  is  provided  should  at 
least  provide  a  basis  for  a  qualitative  assessment  of  the 
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Evaluating  Software  Aspects  of  Systems 


F101.  Software  OT&E  Guidelines.  This  annex  provides  general  guide¬ 
lines  for  OT&E  of  software-intensive  systems  and  computer  software 
subsystems  in  accordance  with  software  initiatives  contained  in  DOD 
Dir  5000.3  and  DOD  Dir  5000.29. 

a.  Background .  Software  is  a  combination  of  computer  programs 
(including  test  and  maintenance  programs),  computer  data,  firmware, 
and  documentation  enabling  computer  equipment  to  perform  various 
computational  or  control  functions .  Firmware  is  a  hardware  com¬ 
ponent  that  obtains  its  functional  characteristics  from  factory- 
fixed  software.  Modern  weapons  systems  use  computers  and  associated 
software  to  perform  functions  critical  to  strategic  and  tactical 
missions.  DOD  estimates  that  over  $3  billion  is  spent  annually  for 
weapons  system  software  and  that  the  co6t  is  steadily  rising,  par¬ 
ticularly  the  cost  of  maintaining  operational  software.  In  addition 
to  increasing  cost,  technical  and  management  problems  with  the  way 
software  is  designed,  developed,  tested,  and  maintained  tend  to 
extend  development  schedules  and  degrade  mission  performance. 

Because  of  lower  visibility  in  the  acquisition  process,  development 
and  testing  of  software  is  not  given  the  same  emphasis  as  hardware, 
even  though  it  is  just  as  critical  to  operational  performance.  Recent 
DOD  software  initiatives  promote  higher  visibility  and  a  more  disci¬ 
plined  approach  to  management  of  software  design,  engineering,  and 
programming  to  ensure  production  of  effective  software  at  minimum 
life-cycle  cost. 

b.  Test  Planning.  Review  DT&E  plans  for  performance  and  quality- 
oriented  testing.  Ensure  that  this  DT&E  is  clearly  defined  in  the 
TEMP  and  that  performance  testing  is  planned  at  the  completion  of 
significant  phases,  particularly  thp  software/hardware  integration 
phase.  Ensure  that  COMOPTEVFOR  is  afforded  the  opportunity  to  parti¬ 
cipate  in  test  design  and  execution.  If  test  results  are  significant, 
be  prepared  to  evaluate  and  report  separately.  Before  computer 
program  acceptance,  the  DA  should  conduct  software  quality  testing. 
This  may  be  the  final  test  of  a  performance  test  series  at  an  LBTS 
(land-based  test  site),  but  if  possible,  it  should  be  completed  in  the 
ultimate  user  environment  (see  MILSTD  1679  and  TADSTAND  9).  If  the 
operational  test  environment  is  realistic  and  testing  is  scenario- 
driven,  valuable  operational  information  can  be  realized. 

c.  Configuration  Management .  During  initial  planning  of  soft¬ 
ware  deveTopment7~rivTivr^onfTguration  management  procedures  to  be 
instituted  during  development.  Development  plans  should  provide 
sufficient  configuration  baselines  to  ensure  stable  software  and 
documentation  before  the  final  IOT&E  test  phases.  COMOPTEVFOR 
should  subsequently  participate  in  analyzing  and  evaluating  the 
operational  impact  of  proposed  changes,  and  whether  they  are  additions 
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to  or  deletions  from  system  software  or  hardware  functions, 

d.  OT-II  Integration  Testing 

(1)  Combined  Testing.  Combined  DT/OT  testing  can  result  in 
significant  cost/time/services  benefits  and  preclude  unnecessary 
duplication.  COMOPTEVFOR  need  not  conduct  a  separate  performance 
or  software  quality  test  on  a  software  subsystem.  However,  indepen¬ 
dent  monitoring  and  evaluation  of  DA  and  contractor  efforts  are 
necessary  to  ensure  that  necessary  operational  issues  have  been  add¬ 
ressed. 


(2)  Scenario-Driven  Tests ♦  As  integration  progresses  and 
tiie  system  matures ,  software  should  be  exercised  in  operational 
scenarios  with  realistic  tactics  that  closely  simulate  the  expected 
combat  environment. 

(3)  Early  Estimates.  If  required  by  higher  authority,  an 
early  estimate  of  potential  system  operational  effectiveness  and 
operational  suitability  may  be  prepared  based  on  hardware  and 
software  performance  at  an  LBTS . 

e.  OT-II  OPEVAL.  System  operational  effectiveness  and  opera¬ 
tional  suitability  will  be  determined  by  testing  the  total  system 
with  fully  integrated  software  and  hardware  in  the  ultimate  user 
environment. 

f .  OT-III/OT-IV 

(1)  Purpose .  OT-II I  is  designed  to  complete  unfinished 
IOT&E,  test  fixes ,  and  refine  tactics.  These  apply  equally  to 
hardware  and  software.  OT-II I  may  be  continued  or  reopened  until 
the  objectives  stated  in  the  TEMP  for  that  phase  have  been  attained. 
OT- IV  is  designed  to  ensure  demonstration  of  the  achievement  of 
program  objectives  for  production  system  operational  effectiveness 
and  operational  suitability.  Other  objectives  include  OT&E  of  the 
system  in  new  environments  or  against  new  threats.  Modifications 
as  a  result  of  fixes  or  in  response  to  new  environments  or  threats 
would  not: 


(a)  Represent  a  major  alteration  of  military  or 
operational  characteristics. 

(b)  Involve  a  hardware  change  requiring  a  major 
production  decision. 


(c)  Be  initiated  by  a  change  in  the  mission  require¬ 


ments  . 

(2)  Fleet  Distribution  Decision.  Since  production 
systems  may  already  be  deployed  and  the  software  fix  or  modifi¬ 
cation  may  result  in  additional  training,  may  affect  interoperability 
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or  may  require  further  tactics  development,  additional  OT&E  may  be 
necessary.  COMOPTEVFOR  should  reassess  the  effectiveness  and 
suitability  of  the  modified  system  in  the  first  shipboard 
installation  and  report  on  that  assessment  before  a  fleet-wide  dis¬ 
tribution  decision  concerning  the  fix  or  modification. 

g.  Significant  Software  Alterations  to  Existing  Systems 

(1)  Significant  Alterations.  The  following  considera¬ 
tions  are  used  to  measure/deteriine  the  significance  of  a  software 
modification: 

(a)  Level  of  funding. 

(b)  New  application  for  which  additional  system 
procurement  is  planned. 

(c)  Generated  hardware  changes  requiring  a  produc¬ 
tion  decision. 

(d)  Effect  on  training. 

(e)  Effect  on  interoperability. 

(f)  Modification  initiated  by  a  change  in  mission 
requirements  altering  the  military  or  operational  characteristics 
of  the  system. 

(2)  T&E  of  Significantly  Modified  systems.  T&E  on 
these  systems  will  b<>  conducted  in  the  same  manner  as  for  new 
systems.  A  T&E  number  will  be  assigned,  a  TEMP  will  be  prepared, 
and  the  system  will  be  operationally  evaluated  to  determine 
operational  effectiveness  and  operational  suitability. 

h.  Summary.  In  general,  COMOPTEVFOR  performs  the  following 
functions  during  software  development  for  new  or  existing  weapons 
systems : 


(1)  Analyzes  and  relates  system  and  software  subsystem 
requirements  to  mission  needs. 

(2)  Monitors  software  development  throughout  by  tracking 
system  operational  requirements  and  identifying  operational  issues. 

(3)  Reviews  OT&E  plans  to  ensure  that  operational  objectives 
have  been  considered. 

(4)  Provides  user-oriented  inputs  or  service  to  DT&E. 

(5)  Evaluates  the  operational  impact  of  major  changes. 
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(6)  Plans  and  conducts  OPEVAL  on  the  new  or  modified  system. 
F102.  Software  Development  Phases  and  Milestones 

a.  Phases .  Software  is  a  paper  product.  There  are  no  develop¬ 
ment  models  or  prototypes  to  provide  visible  milestones.  The  phases 
are  marked  by  their  related  documents . 

(1)  Requirements  Phase.  System  requirements  are  normally 
partitioned  between  software  and  hardware.  This  division  results 
in  two  types  of  documents  that  represent  this  phase: 

(a)  Program  Performance  Requirements.  This  document 
states  the  required  contribution  of  software  to  system  performance . 
This  statement  must  be  complete,  correct,  and  traceable  to  mission 
needs.  It  must  be  reviewed  against  system  requirements  for  complete¬ 
ness. 


(b)  Interface  Design  Requirements ♦  Interface  requirements 
coordinate  the  activity  of  the  hardware  engineering  effort  to  the 
software  subsystem  by  ensuring  adherence  to  key  specifications  so 
that  the  software  may  work.  This  document  or  documents  may  also 
describe  how  a  particular  software  subsystem  relates  to  other  systems. 

(2)  Program  Design  Phase.  This  phase  entails  functional 

a? ' ecation  of  tasks  to  be  performed  by  subprograms  and  their  modules . 
Memory  and  timing  budgets  are  laid  out.  Often  a  design  walkthrough 
is  performed,  and  the  phase  is  terminated  by  a  PDR  (Preliminary 
Design  Review).  The  basic  document  is  the  Program  Design  Require¬ 
ments,  a  written  procedural  representation  of  what  the  software 
subsystem  is  to  do  in  some  combination  of  English,  flow  charts, 
and  program  design  language.  The  OTD  should  attend  the  PDR. 

(3)  Module  Implementation  and  Coding  Phase.  The  project  is 
broken  into  many  parallel  mini-projects.  A  CDR  (Critical  Design 
Review)  is  often  performed  early  in  this  phase.  The  output  of  this 
process  is  a  set  of  debugged  software  modules  that  nm  correctly 
by  themselves.  The  OTD  should  also  attend  the  CDR. 

(4)  Integration  and  Testing  Phase.  Historically,  one-third 
to  one-half  of  calendar  project  ti ite  snd  man-hours  are  expended  in 
this  phase.  The  output  of  this  phase  is  a  single  system  that  works 
and  conforms  to  system  requirements. 

(a)  Software/Software  Integration.  Individual  modules 
are  combined  and  tested  to  form  working  subsystems. 

(b)  Software/Hardware  Integration  and  Testing.  The 
software  subsystem  is  integrated  with  the  results  of  hardware 
development.  This  process  is  often  performed  at  an  LBTS  and  may 
be  expensive  and  time-consuming. 
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b.  Milestones.  Milestones  in  a  software  project  are  typically 
document  reviews.  The  broad  phases  are  separated  or  highlighted 
by  documents.  Milestones  are  events,  not  a  percentage,  and  the 
milestones  must  be  successfully  passed  (i.e.,  the  document  complete 
and  approved)  to  mark  the  end  of  a  phase.  Some  significant  mile¬ 
stones  that  an  OTD  may  become  involved  with  are  listed  below  by 
phase  and  generic  name: 

(1)  Requirement  Phase 

(a)  Mission  Requirements  Review. 

(b)  System  Requirements  Review. 

(c)  Software  Requirements  Review. 

(d)  System  Performance  Test  Requirements  Review. 

(2)  Program  Design  Phase.  Functional  Software  Design 
Review  ( PDR ) . 


(3)  Module  Implementation  and  Coding  Phase. 
Module  Design  Review  (CDR). 

(4)  Integration  and  Testing  Phase 


Detailed 


(a)  Program  Certification. 

(b)  Software  Quality  Test. 

(c)  Program  Acceptance. 

F103.  Performance  Measures  and  Analysis 

a.  General .  Analysis  procedures  in  test  planning,  test 
execution,  and  reporting  for  systems  with  major  software  subsystems 
are  very  similar  to  those  procedures  for  systems  without  software. 

For  most  of  the  analysis,  software  need  not  be  considered  separately 
from  the  hardware,  in  this  sense,  software  is  simply  an  internal 
system  component  that  gives  the  hardware  system  its  particular 
external  characteristics.  Firmware  should  be  treated  as  software 
until  installation  in  a  production  system.  When  installed  in  a 
production  system,  the  OTD  should  consider  firmware  as  a  piece  of 
hardware.  The  trend  in  hardware  development  is  towards  increased 
use  of  software  and/or  firmware  since  software  tends  to  increase 
system  flexibility  and  lower  modification  costs.  In  this  context, 
software  is  a  means  to  an  end  rather  than  an  end  in  itself.  The 
measure  of  mission  success  methodology  is  just  as  valid  for  software 
intensive  projects  as  for  hardware  only  projects:  Scenario-driven 
testing  of  complete  systems  is  still  the  key  principle  for  system" 
evaluation,  while  reporting  effaptivanaan  anHsuitSbiTT^  ox  total 
system  is  still  the  prime  requirement.  While  the  general  analysis 
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procedures  are,  for  the  most  part  unchanged,  the  critical  impact 
of  software  does  require  some  additional  definitions  and  measures, 
especially  in  the  area  of  failure  analysis.  These  additional  measures 
arise  out  of  unique  life-cycle  problems  associated  with  software, 
and  are  used  to  aid  in  subsequent  repair  of  software  faults.  These 
software  measures  and  their  application  are  explained  in  the  para¬ 
graphs  that  follow.  Note  however,  that  the  measures  of  effective¬ 
ness  that  are  determined  for  the  software  alone  are  used  mostly  for 
the  DA  to  correct  reported  failures,  whereas  only  overall  system 
effectiveness/suitability  should  be  used  to  determine  readiness  for 
ASU. 


b-  Effectiveness  Criteria.  The  formation  of  operational 
requirements  for  various  missions  is  still  based  on  operational 
need.  Therefore,  in  operational  effectiveness  testing  it  is 
unnecessary  (and  in  most  cases  impossible)  to  set  separate  goals 
for  software,  although  the  requirement  remains  to  measure  and  report 
the  effects  of  software  on  system  performance.  Overall  system  opera¬ 
tional  effectiveness  is  the  primary  item  to  test  for  and  report  on; 
report  the  effects  of  software  on  operational  effectiveness  only  if 
they  can  be  isolated. 

c.  Suitability  Criteria.  In  the  suitability  area,  separating 
the  software  effects  on  the  system  will  provide  a  more  meaningful 
evaluation  and  will  help  the  DA  fix/improve  the  system;  they  should,  ( 
therefore,  be  measured  when  possible.  Remember,  system  thresholds 
specify  the  operational  requirements  of  the  entire  system  and  not 
components.  They  pertain  to  both  hardware  and  software.  Separate 
threshold  values  for  unique  combinations  of  hardware  and  software  may 
be  presented  as  part  of  the  analysis  and  reporting  effort.  Note, 
however,  that  measuring  and  reporting  separately  does  not  reduce  the 
need  to  concentrate  on  determining  the  characteristics  of  the  total 
system  as  the  prime  requirement.  Additionally,  human  factors  associa¬ 
ted  with  the  operator  interface  to  the  system  (displays,  control  func¬ 
tions,  etc  )  are  highly  important.  For  example,  an  aircraft  pilot  may 
indicate  that  if  certain  data  were  reformatted  or  moved  to  another 
display,  system  effectiveness  would  be  increased. 

d.  Early  Testing.  The  focus  of  attention  at  OT-O  and  OT-I 
should  be  directed  toward  assessment  of  the  system's  functions 
and  how  they  support  its  concept  of  operations.  This  means  that 
assessment  of  software  design,  and  internal  organization  and 
operation,  should  be  left  to  the  DA.  Early  OT-I I  should  be  directed 
toward  correcting  gross  errors  discovered  during  OT-I  or  during  the 
early  specification  efforts  of  OT-II.  The  object  is  to  validate  the 
software  and  hardware  design  on  the  basis  of  how  well  the  implemented 
functions  accomplish  the  system's  operational  requirements.  In 
summary,  early  testing  should  focus  on  either  the  integrated  program 
at  the  LBTS  or,  if  available,  breadboard  models. 
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e.  OT-II  OPEVAL.  At  this  stage,  it  is  expected  that  easily 
isolated  software  problems  have  been  corrected.  Because  of  the 
impracticality  of  testing  the  almost  infinite  number  of  possible 
program  paths  in  early  OT&E,  software  problems  can  still  be 
expected  during  OPEVAL.  Most  remaining  software  problems  are  due 
to  use  of  faulty  paths  not  previously  executed.  Bowever,  testing 
in  scenario  fashion  leads  to  the  assumption  that  use  of  these 
faulty  paths  is  random,  and  use  of  a  constant  failure  rate  for 
analytical  purposes  is  feasible. 

(1)  Definitions 

(a)  Hardware  Fault.  Any  fault  clearly  associated 
with  hardware,  such  as  power  loss,  broken  CRT,  no  video,  etc. 

(b)  Software  Fault.  Any  fault  in  which  the  hardware 
appears  operational  but  the  system  is  not  functioning  as  required. 

(2)  Reliability  (Test  S-l) 

(a)  Although  technical  agencies  define  failures  based 
on  MIL  STD  1679  and  TADSTAND  9,  operational  failures  are  defined 
in  terms  of  whether  or  not  mission  abort/degradation  could  have 
occurred  or  did  occur.  These  operational  definitions  will  be 
defined  not  only  for  the  total  system  but  also  for  hardware  and 
software  separately.  For  hardware,  reliability  computations  are 
based  on  the  assumption  that  failures  are  uniformly  distributed 

in  time  (reliability  estimates  follow  an  exponential  distribution). 
For  software,  this  assumption  is  not  technically  correct  but  can 
be  used  provided  the  following  conditions  are  met: 

1.  Major  software  failures  (causing  the  system 
to  be  totally  inoperative  or  marginal  on  a  continuous  basis)  have 
been  corrected  before  OPEVAL. 

2.  Minor  software  failures  that  are  consistent 
(reproducible)  are  corrected  before  OPEVAL  or  are  worked  around 
during  testing.  For  example,  if  a  certain  sequence  of  operator 
interactions  is  known  to  produce  a  fault  but  could  be  avoided  by 
modifying  the  sequence,  these  faults  should  not  be  included  in 
reliability  computations  but  should  be  reported  and  discussed  as 
they  relate  to  overall  system  operational  effectiveness . 

3.  The  software  system  is  large  (in  the  sense  of 
many  functions ) . ” 

(b)  If  the  above  conditions  are  met,  the  MTBF  for 
software  is  determined  by  dividing  total  system  running  time 
(under  operational  stress)  by  the  total  number  of  mission  aborting/ 
mission  degrading  software  faults  (critical  or  major  failures). 
Also,  MTBF  for  hardware  is  computed  as  defined  in  COMOPTEVFORIHST 
3960.7,  Analyst  Notebook.  Finally,  total  system  MTBF  is  total 
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system  uptime  (under  stress)  divided  by  the  sum  of  hardware 
failures  and  software  faults.  Using  the  MTBF  estimates,  the 
probability  of  the  system  operating  a  specified  length  of  time 
can  be  computed  from  the  exponential  distribution.  In  some 
projects,  the  system  tuider  test  has  more  than  one  mode  of 
operation  or  mission,  so  it  is  important  to  report  the  reli¬ 
ability  of  each  mode  or  mission,  rather  than  one  overall  figure. 
Data  must  be  carefully  analyzed  to  insure  that  they  are  correctly 
applied  to  each  separate  mode  or  mission.  Failure  rates  are  to 
be  computed  for  each  mode  of  operation  to  produce  a  weighted 
software  system  failure  rate. 

(c)  Definition.  Software  reliability  is  the  probability 
that  the  software  subsystem  will  operate  a  specified  period  of 
time  under  given  environmental  conditions  without  a  mission 
aborting/system  degrading  software  fault  (critical  or  major 
failure ) . 

(3)  Maintainability  (Test  S-2).  Separation  of  hardware 
and  software  for  analysis  Decodes  particularly  useful  in  this 
area.  Hardware  failures  lead  to  fault  locate  time,  supply  problems, 
replacement  time,  calibration  time,  etc.  The  MTTR  may  be  in 

hours  or  days.  Software  mean  restoration  time  may  be  a  matter  of 
seconds  or  minutes  (with  no  logistics  problems).  However,  time 
to  restore  must  include  time  to  restore  the  software  data  base 
and  files  to  their  state  before  the  failure,  which  may  extend 
restoration  times  significantly.  For  example,  a  fire  control 
system  loses  all  track  data  if  the  system  must  be  restarted. 

These  track  data  must  be  reentered  manually  or  new  tracks  must  be 
established.  This  recovery  time  should  be  included  in  MTTR 
computations.  Note  that  the  point  in  time  when  a  system  is  fully 
operational  may  be  a  judgement  call  by  the  OTD  given  the  above 
restorations  characteristics . 

(4)  Availability  (Test  S-3).  System  downtimes  caused  by 
software  failures  are  normally  terminated  by  program  reloads  or 
restarts.  Program  reloads  do  not  have  the  typical  hardware  type 
of  extended  downtimes  caused  by  fault  isolation  time,  supply 
response  time,  etc.  Therefore,  availability  figures  for  software 
are  calculated  using  a  relatively  short  downtime  and  result  in  A 
numbers  very  close  to  unity.  With  A  normally  close  to  1,  soft-0 
ware  availability  becomes  an  insignificant  measure. 

(5)  Qualitative  Suitability  Tests.  Except  for  parts 
control,  the  qualitative  approach  and  measures  in  Test  S-4  apply 
to  software  as  to  hardware.  Again,  the  emphasis  is  on  treating 
software  as  a  subsystem  for  purposes  of  analysis.  The  life-cycle 
testing  described  in  MIL  STD  1679  is  not  a  part  of  OPEVAL;  these 
tests  must  be  conducted  before  OPEVAL  unless  a  waiver  is  granted 
by  OP-098.  Human  factors,  training,  and  interoperability  are  key 
tests  and,  except  for  human  factors,  follow  the  same  patterns  as 
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for  hardware.  Human  factor  tests  for  software  intensive  systems 
require  special  treatment  as  outlined  in  the  next  paragraph. 

(6)  Human  Factors  Tests.  The  hardware- related  tests  for 
software-intensive  systems  are  still  required,  but  particular 
attention  should  be  paid  to  the  following  major  categories 
(refer  to  Question  Catalog  for  Computer  Generated  Questionnaire): 

(1)  Displays;  (2)  Workspace;  (3)  Controls;  (4)  Training;  (5) 
Documentation;  (6)  Operating  Procedures;  (7)  Life  Cycle  Support. 

In  addition,  strictly  software  questions  should  be  asked  (see 
computers  and  software  in  Questionnaire  Catalog  as  amended). 

(a)  Operator-System  Interface.  The  primary  inter¬ 
action  between  the  operator  and  the  software  subsystem  is  through 
the  system  workstation.  This  workstation  normally  includes  some 
of  the  following:  a  CRT  display;  various  hardcopy  devices 
(printers,  BTRs,  LOFAR  grams,  etc);  a  keyboard;  a  button  panel; 
status  lights;  and,  occasionally,  audible  alarms.  The  efficiency 
with  which  the  operator  interacts  with  these  input/output  devices 
(and  thereby  the  software  subsystem)  can  determine  the  success  or 
failure  of  the  mission  during  a  high  pressure  (threat)  environ¬ 
ment.  Therefore,  problems  associated  with  the  operator  interface 
must  be  examined  carefully.  The  basic  question  is  whether  the 
operator  has  available  all  the  information  required  to 
accomplish  his  task  in  the  required  time  frame  and  in~a  format 

he  can  use  efficiently.  In  other  words,  is  the  mission  degraded 
because  of  the  information  exchange  between  the  operator  and  the 
software  subsystem? 

(b)  IPL  (Initial  Process  Load).  Initial  loading  of 
computer  software  is  frequently  a  time-consuming  and  error-prone 
evolution.  Questions  should  be  asked  to  evaluate  the  reliability 
and  facility  of  IPL. 

(c)  Diagnostics.  Most  hardware  and  software  systems 
provide  either  automated  or  operator-assisted  diagnostics  to 
isolate  and  report  on  system/failures.  This  is  absolutely  essential 
for  large  scale  software  systems.  Also,  the  system  must  provide 

a  software  test  capability  to  determine  the  go/no-go  condition 
of  the  software.  The  availability  of  such  tools  and  the  ease 
with  which  they  are  used  will  have  a  significant  effect  on  system 
availability. 

F104.  Software  Issues  Checklist.  The  following  list  of  items 
nay  or  may  not  apply  to  a  particular  program.  They  are  listed  by 
test  phase  and  are  not  in  priority  order.  Their  intention  is 
mental  stimulation  of  the  OTD  during  program  review. 

a.  OT-O  and  OT-I 

(1)  Have  operator  personnel  been  involved  in  the  planning 
and  design  of  the  system  software? 
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(2)  Have  software  requirements  for  operator  overide/ 
lock-out  control  features  been  evaluated? 

(3)  Has  the  hardware/software/operator  functional  require¬ 
ment  mix  been  evaluated? 

(4)  Have  the  intended  functions  of  the  program  been 
clearly  specified,  and  do  they  support  the  system  operational 
requirements  ? 

(5)  Have  procedures  for  software  management  (changes  and 
improvements)  been  developed? 

(6)  Have  software  management  procedures  been  published  in 
a  Software  Management  Plan/Software  Life  Cycle  Management  Plan/ 
Computer  Resources  Plan  or  Software  Development  Plan? 

(7)  Have  quantitative  and  demonstrable  performance 
objectives  been  established  and  recorded  in  the  NDCP  and  TEMP? 

(8)  Have  software  requirements  been  defined  before 
Milestone  II? 

(9)  Has  the  Life  Cycle  Support  Agency  been  assigned? 

(10)  Do  the  support  software,  simulators,  and  training 
modules  exist,  or  are  they  being  developed  and  tested? 

(11)  Has  the  LBTS  been  chosen  or  planned? 

(12)  Have  programming  personnel  been  involved  in  the 
testing  process?  (Build  a  little,  test  a  little. ) 

(13)  Have  core  size  estimates  been  established? 

(14)  Is  there  room  in  core  to  expand  the  program?  The 
data  base?  (As  required  in  TADSTAND  D.) 

(15)  Has  a  stability  estimate  for  each  functional  require¬ 
ment  been  established?  (Sensitivity  to  operational  environment 
and  tactics . ) 

(16)  Does  the  development  contract  allow  for  change? 

(17)  Have  all  interface  requirements  been  established  (to 
other  digital  systems,  to  (subsystems,  to  user,  to  other  Services) 
for  each  system  mode  including  OFF? 

(18)  Have  performance  objectives  been  established  for 
casualty  mods  operations? 
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(19)  Have  on-line/off-line  fault  locate  provisions  been 
included  in  the  system  design? 

(20)  Has  an  independent  V&V  (verification  and  validation) 
agent  and  procedures  been  established? 

(21)  Has  vital  information  that  must  be  protected  from 
failure  been  declared? 

(22)  Has  the  testing  been  designed  to  discover  errors, 
not  to  prove  that  the  software  is  error  free? 

(23)  Are  errors  being  recorded?  Corrected  promptly? 

Early  failure  rates  analyzed? 

(24)  Is  testing  being  designed  to  cover  a  broad  range  of 
stressful  conditions,  including  integrated  full  system  employment? 

b.  OT-II  Integration 

(1)  Are  software  data  transfer  requirements  clear? 

(2)  What  data  are  lost  following  a  halt  and  restart,  and 
how  long  does  it  take  to  recover  or  regenerate  the  operating  data 
base? 

(3)  Have  configuration  management  procedures  been 
established?  Being  followed?  Are  effective? 

(4)  Are  changes  to  the  system  being  documented?  Formally 
approved? 

(5)  Have  firm  procedures  for  core  reserve  management  been 
established?  Being  followed?  (TADSTAND  D  again. ) 

(6)  Does  the  software  subsystem  continue  to  reflect 
system  operational  requirements? 

(7)  Are  current  or  projected  tactical  publications/operational 
environment/tactics  being  implemented  in  software  design?  (Assess 

at  PDR  and  CDR. ) 

(8)  Has  a  document  review  been  conducted?  (PDR  and  CDR.) 

(9)  is  the  Life  Cycle  Support  Agency  participating  in 
Integration  Testing? 

c.  OT-II  OPEVAL 

(1)  Is  the  program  documented  and  by  what  standard? 
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(2)  Before  OPEVAL,  have  criteria  and  rules  for  failure 
assignment  to  components  and  subsystems  been  firmly  established? 

(3)  Has  the  software  demonstrated  a  level  of  maturity 
sufficient  to  proceed  to  OPEVAL?  (TADSTAND  9  testing  complete?) 

(4)  Has  test  design  provided  procedures  to  verify  correction 
of  previous  errors/operational  issues? 

(5)  Does  the  ILSP  include  software  maintenance  procedures? 

(6)  Can  software  failures  be  correlated  to  events/operational 
environment?  Can  the  stress  boundary  be  identified  or  defined? 

(7)  Does  the  evaluation  report  contain  an  assessment  of 
software  maintainability? 

d.  OT-III  and  OT-IV 

(1)  Has  the  change  control  and  change  management  system 
been  kept  in  effect  until  all  objectives  of  the  TEMP  are  attained 
and  the  T&E  number  is  cancelled? 

(2)  Has  conversion  to  the  Life  Cycle  Support  Agency  been 
effected  in  accordance  with  the  ILSP? 

(3)  Has  the  system  been  designated  for  new  applications 
resulting  in  new  interface  requirements? 

(4)  Has  all  IOT&E  been  finished? 

(5)  Hav i  effective  procedures  for  responding  to  fleet 
software  problems  been  implemented? 
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Annex  G 

Assessing  Logistic  Supportability 


G101 .  Introduction 


(R 


a.  As  its  name  implies,  the  ILS  (Integrated  Logistic  Support) 

Plan  is  the  document  in  which  the  DA  discusses  the  various  methods 
it  will  use  to  provide  the  full  range  of  logistic  support  for  the 
system  in  question.  The  ILS  Plan  should  specify  what  support  tasks 
will  be  accomplished,  who  will  be  responsible  for  their  accomplish¬ 
ment,  and  how  and  when  they  will  be  accomplished. 

b.  ILS  planning  is  an  iterative  process  that  begins  in  the  pro¬ 
gram  initiation  phase  (before  Milestone  I)  and  becomes  increasingly 
more  specific  throughout  the  acquisition  cycle.  The  same  applies 

to  the  ILS  Plan;  the  degree  of  detail  required  in  it  is  a  function 
of  the  system's  status  in  the  acquisition  cycle. 

c.  The  checklist  provided  in  this  Annex  is  a  guideline  for  con¬ 
ducting  a  meaningful  review  of  any  ILS  Plan  —  although  aviation 
systems  use  somewhat  different  terminology.  Not  all  support  topics 
will  always  apply;  the  nature  of  the  system  and  its  level  of  maturity 
will  determine  whether  or  not  any  given  aspect  of  support  should  be 
addressed  in  any  given  ILS  Plan.  Nevertheless,  even  ILS  Plans  for 
systems  in  the  program  initiation  phase  should  reflect  an  under- 
standing  of  the  full  range  of  logistic  support  considerations.  Such 

TcJ?  an  understanding  is  expressed  by  including  schedules  for  completion 
of  support- related  tasks  (e.g.,  conducting  the  provisioning  con¬ 
ference,  finalizing  the  interim  support  plan,  completing  preliminary 
technical  manuals,  identifying  support  and  test  equipment)  even 
though  the  exact  nature  of  these  tasks  may  not  yet  be  known. 

d.  The  OTD/OTC  should  use  his  knowledge  of  the  system  to  identify 
those  aspects  of  ILS  that  are  not  adequately  addressed  in  the  ILS 
Plan.  Deficiencies  should  be  discussed  with  the  DA  at  the  earliest 
opportunity,  and  should  be  reviewed  for  potential  inclusion  in  a 
future  Evaluation  Report.  The  most  significant  recurring  deficien¬ 
cies  in  ILS  Plans  are: 


(1)  Submission  of  PTD  (provisioning  technical  documentation) 
to  the  SPCC  (Ships  Parts  Control  Center)  is  not  included  as  an  ILS 
milestone,  or  the  milestone  is  set  too  far  in  the  future.  (Navy 
supply  support  will  not  be  available  in  less  them  18  months  after 
PTD  submission. ) 

(2)  Planning  for  the  interim  (contractor)  support  period  is 
inadequate . 

(3)  Depot  level  maintenance  activities  are  not  designated; 
when  depot  level  responsibilities  will  shift,  there  is  inadequate 
planning  for  the  transition. 
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(4)  No  milestone  is  established  for  identifying  all  common 
and  special  test  equipment. 

(5)  Support  of  test  equipment  (e.g.,  training  and  operation/ 
maintenance  manuals)  is  not  considered. 
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ILS  Plan  Checklist 


I .  ILS  Program  Planning 

a.  Are  appropriate  organizational  charts,  responsibility  matrices 
ILSMT  (ILS  Management  Team)  membership,  and  related  ILS  Program 
organizational  structures  provided? 

b.  Are  SPCC  representatives  included  as  members  of  the  ILSMT? 

c.  Is  the  process  for  revising  the  ILS  Plan  described? 

d.  Are  completion  dates  set,  and  responsible  agencies  designated, 
for  the  following  ILS  inputs? 

(1)  Navy  Training  Plan  Conference. 

(2)  Navy  Training  Plan. 

(3)  Navy  Support  Date. 

(4)  Provisioning. 

(a)  Issuance/funding  of  a  PRS  (Provisioning  Requirements 

Statement ) . 

(b)  Submission  of  PTD  by  the  contractor,  or  certification 
of  PTD  before  submission. 

(c)  Submission  of  an  SML  (Support  Material  List)  by  the 

contractor . 


(d)  Funding  and  ordering  of  SML  spare  parts  by  the  DA. 

(e)  Assignment  of  Source,  Maintenance,  and  Recoverability 

Codes . 


(f)  Delivery  of  initial  OBRP  (on-board  repair  parts)  to 
each  applicable  maintenance  activity. 

I I .  Maintenance 


a.  Has  an  LSA  (logistic  support  analysis)  effort  been  implemented 
in  accordance  with  MIL-STD-1388,  1388-1,  and  1388-2?  If  not,  is 

a  justification  for  not  conducting  an  LSA  provided? 

b.  Is  the  extent  to  which  LORA  (level  of  repair  analysis)  will 
be  applied  addressed?  (MIL-STD-1390  applies.) 

c.  Are  ship  installation,  the  method  of  installation,  and  check¬ 
out  planning  and  schedule  information  provided? 
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d.  Are  the  three  levels  of  maintenance  —  0  (organizational), 

I  (intermediate),  and  D  (depot)  —  discussed,  even  though  not  all 
are  used? 

e.  Are  reasons  given  for  any  levels  of  maintenance  that  are 
not  used? 

f.  If  required,  have  I-  and  D-level  maintenance  activities  been 
designated  and  their  specific  tasks  delineated? 

g.  Are  the  expected  workloads  for  corrective  and  preventive 
maintenance  defined  for  each  level  of  maintenance? 

h.  If  I-  or  D-level  maintenance  is  initially  performed  by  the 
contractor,  are  adequate  provisions  made  for  transition  to  organic 
support?  (The  timing  and  funding  of  the  transition  should  be 
addressed. ) 

i.  Activities  designated  to  perform  each  level  of  maintenance 
will  frequently  change  over  the  program's  life  (from  DT&E  to  OPEVAL 
to  service  use).  Are  these  shifts  presented  in  the  ILS  Plan  so  that, 
at  any  point  in  time,  the  agency  responsible  for  each  set  of  main¬ 
tenance  requirements  can  be  easily  identified? 

j.  If  new  facilities  (e.g.,  shops,  buildings,  maintenance  areas) 
are  required,  will  they  be  available  to  support  the  system? 
(Typically,  military  construction  programs  require  5  to  6  years 

for  budgeting  and  completion. ) 

k.  Are  special  skills  and  the  numbers  of  technicians  in  each 
skill  category  adequately  defined  at  each  level  of  maintenance? 

l.  Is  maintenance  of  support  and  test  equipment  considered? 
(APL/AEL  (Allowance  Parts  List/Allowance  Equipage  List)  support 
requirements,  calibration  requirements,  etc.  should  be  addressed.) 

m.  If  applicable,  have  provisions  been  made  for  software  main¬ 
tenance?  Have  imbedded  systems  been  identified? 

Ill .  Supply  Support 

a.  Is  the  supply  support  for  Training  Units  the  same  as  for 
shipboard  installations? 

b.  Given  the  thresholds  established  for  reliability  and  opera¬ 
tional  availability,  is  the  allowable  mean  logistic  delay  realistic? 

c.  Are  initial  spare  parts  procurements  scheduled  so  as  to  sup¬ 
port  the  plans  for  production  and  deployment  and  the  Navy  Support 
Date? 
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ILS  Plan  Checklist 


I .  ILS  Program  Planning 

a.  Are  appropriate  organizational  charts,  responsibility  matrices, 
ILSMT  (ILS  Management  Team)  membership,  and  related  ILS  Program 
organizational  structures  provided? 

b.  Are  SPCC  representatives  included  as  members  of  the  ILSMT? 

c.  Is  the  process  for  revising  the  ILS  Plan  described? 

d.  Are  completion  dates  set,  and  responsible  agencies  designated, 
for  the  following  ILS  inputs? 

(1)  Navy  Training  Plan  Conference. 

(2)  Navy  Training  Plan. 

(3)  Navy  Support  Date. 

(4)  Provisioning. 

(a)  Issuance/funding  of  a  PRS  (Provisioning  Requirements 

Statement) . 

(b)  Submission  of  PTD  by  the  contractor,  or  certification 
of  PTD  before  submission. 

(c)  Submission  of  an  SML  (Support  Material  List)  by  the 

contractor . 


(d)  Funding  and  ordering  of  SML  spare  parts  by  the  DA. 

(e)  Assignment  of  Source,  Maintenance,  and  Recoverability 

Codes . 


(f)  Delivery  of  initial  OBRP  (on-board  repair  parts)  to 
each  applicable  maintenance  activity. 

II .  Maintenance 


a.  Has  an  LSA  (logistic  support  analysis)  effort  been  implemented 
in  accordance  with  MIL-STD-1388,  1388-1,  and  1388-2?  If  not,  is 

a  justification  for  not  conducting  an  LSA  provided? 

b.  Is  the  extent  to  which  LORA  (level  of  repair  analysis)  will 
be  applied  addressed?  (MIL-STD-1390  applies.) 


c.  Are  ship  installation,  the  method  of  installation,  and  check¬ 
out  planning  and  schedule  information  provided? 
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d.  Are  the  three  levels  of  maintenance  —  0  (organizational), 

I  (intermediate),  and  D  (depot)  —  discussed,  even  though  not  all 
are  used? 

e.  Are  reasons  given  for  any  levels  of  maintenance  that  are 
not  used? 

f.  If  required,  have  I-  and  D-level  maintenance  activities  been 
designated  and  their  specific  tasks  delineated? 

g.  Are  the  expected  workloads  for  corrective  and  preventive 
maintenance  defined  for  each  level  of  maintenance? 

h.  If  I-  or  D-level  maintenance  is  initially  performed  by  the 
contractor,  are  adequate  provisions  made  for  transition  to  organic 
support?  (The  timing  and  funding  of  the  transition  should  be 
addressed. ) 

i.  Activities  designated  to  perform  each  level  of  maintenance 
will  frequently  change  over  the  program's  life  (from  DT&E  to  OPEVAL 
to  service  use).  Are  these  shifts  presented  in  the  ILS  Plan  so  that, 
at  any  point  in  time,  the  agency  responsible  for  each  set  of  main¬ 
tenance  requirements  can  be  easily  identified? 

j.  If  new  facilities  (e.g.,  shops,  buildings,  maintenance  areas) 
are  required,  will  they  be  available  to  support  tne  system? 
(Typically,  military  construction  programs  require  5  to  6  years 

for  budgeting  and  completion. ) 

k.  Are  special  skills  and  the  numbers  of  technicians  in  each 
skill  category  adequately  defined  at  each  level  of  maintenance? 

l.  Is  maintenance  of  support  and  test  equipment  considered? 
(APL/AEL  (Allowance  Parts  List/Allowance  Equipage  List)  support 
requirements,  calibration  requirements,  etc.  should  be  addressed.) 

m.  if  applicable,  have  provisions  been  made  for  software  main¬ 
tenance?  Have  imbedded  systems  been  identified? 

Ill .  Supply  Support 

a.  I 8  the  supply  support  for  Training  Units  the  same  as  for 
shipboard  installations? 

b.  Given  the  thresholds  established  for  reliability  and  opera¬ 
tional  availability,  is  the  allowable  mean  logistic  delay  realistic? 

c.  Are  initial  spare  parts  procurements  scheduled  so  as  to  sup¬ 
port  the  plans  for  production  and  deployment  and  the  Navy  Support 
Date? 
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d.  Does  the  ILS  Plan  state  how  repair  parts  will  be  provided 
to  I -level  maintenance  activities? 

e.  Is  the  submission  of  all  PTD  (MIL-STD-1522  applies)  to  the 
spec  scheduled  at  least  18  months  before  the  Navy  Support  Date? 

f .  Are  spares  to  be  managed  as  ready  spares  (i.e.,  stowed  in 
divisional  spaces,  but  still  under  the  cognizance  of  the  Supply 
Department)  so  designated?  Is  the  number  of  these  things  held  to 
a  minimum,  to  facilitate  collection  of  usage  data  by  the  Supply 
Department? 

g.  Preliminary  Supply  Support 

(1)  Is  the  exact  duration  of  the  contractor  support  period 
defined?  Are  provisions  included  for  extending  this  period  if  the 
Navy  Support  Date  is  delayed? 

(2)  Does  the  ILS  Plan  provide  details  for  transition  from 
contractor  to  organic  support? 

(3)  Are  the  types  and  quantities  of  INCO  (installation  and 
checkout)  kits  described?  Are  provisions  made  for  their  timely 
delivery? 

(4)  Does  the  ILS  Plan  state  whether  the  contractor  must 
supply  all  required  spare  parts  during  the  interim  support  period, 
or  merely  a  specified  liBt  of  items? 

(5)  Are  contractor  performance  criteria  (e.g.,  supply  res¬ 
ponse  time  and  turnaround  times  for  spares  to  be  repaired  com¬ 
mercially)  specified? 

(6)  Are  CFE/GFE  (contractor/government  furnished  equipment ) 
included  in  the  interim  support  plan? 

(7)  Are  procedures  described  for  requisitioning  material 
from  non-Navy  sources  during  OPEVAL  and  the  interim  support  period? 

(8)  In  view  of  the  potential  problems  associated  with  dis¬ 
ruption  of  supply  routines,  are  all  non-standard  procedures  for 
acquisition/expenditure  of  spares  justified? 

(9)  Are  procedures  described  for  expediting  material  and 
for  providing  requisition  status  during  the  interim  support  period? 

h.  The  expense  associated  with  a  repairable  system  requires 
that  this  aspect  of  supply  support  be  afforded  considerable  detail 
in  the  ILS  Flan.  Ask  the  following  questions: 

(1)  Is  a  designated  overhaul  point  assigned  for  each  repair- 
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able  to  be  repaired  at  the  depot  level?  (Existing  facilities  should 
be  employed  to  the  greatest  extent  possible.) 

(2)  Are  the  source  of  the  core  pool  of  equipment  and  the 
responsibility  for  funding  the  spares  pool  discussed? 

(3)  Are  sufficient  details  provided  to  cover  shipment  of 
failed  repairables  to  the  appropriate  I-  and  D-level  activities? 
(Shipping  address,  required  documentation,  identification  of  repair¬ 
able  material  (especially  important  for  non-standard  items),  and 
instructions  concerning  limits  on  cannibalization  at  the  O-level 
should  all  be  addressed.) 

(4)  Is  the  packaging  required  by  ship's  force  to  return 
failed  repairables  to  the  Designated  Overhaul  Point  described  by, 
and  provided  for  in,  the  ILS  Plan? 

IV.  Support  and  Test  Equipment 

a.  Are  all  special  tools,  as  well  as  peculiar  and  common  sup¬ 
port  and  test  equipment,  identified  that  are  required  at  each  level 
of  maintenance? 

b.  Are  there  provisions  to  ensure  that  all  items  identified  in 
IV. a.  above  will  be  made  available  in  a  timely  manner?  (Plans  for 
procuring  and  delivering  any  peculiar  support  and  test  equipment  not 
yet  fully  developed  should  be  addressed  specifically. ) 

c.  Are  the  method  and  periodicity  of  test  equipment  calibra¬ 
tion  discussed? 

d.  Is  procurement/delivery  of  auxiliary  pieces  of  special  test 
equipment  (connectors,  cables,  chart  paper,  etc.)  discussed? 

e.  where  applicable,  have  arrangements  been  made  to  modify  test 
equipment  software  in  conjunction  with  software  changes  in  opera¬ 
tional  equipment? 

v.  Packaging,  Handling,  Storage,  and  Transportation 

a.  Are  all  required  special  containers,  lifting  rigs,  and  dol¬ 
lies  scheduled  for  delivery? 

b.  Are  special  problems  and  solutions  during  underway  replenish¬ 
ment  addressed?  Are  all  applicable  methods  of  underway  replenishment 
addressed? 

c.  Have  special  containers  and  special  handling  equipment  been 
validated? 

d.  Where  applicable,  are  special  precautions  described  for 
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^sensitive  items  such  as  wet-cell  batteries  and  integrated  circuit 
y boards? 

e.  Is  packing  material  used  in  accordance  with  safety  regula¬ 
tions? 

f.  Are  special  preservation  requirements  and  shelf  lives  indi 
cated? 

VI.  Technical  Logistic  Data 


a.  Has  the  ILS  Plan  provided  the  following  information/pro¬ 
cedures  in  accordance  with  NAVSEAINST  4105.1? 

(1)  Technical  manual  development  and  maintenance. 

(2)  Change  control. 

(3)  Acceptance  planning. 

(4)  In-process  review. 

(5)  Validation  and  verification. 

b.  will  draft  versions  of  technical  manuals  and  PUS  (Planned 
Maintenance  System)  documentation  be  provided  to  COMDPTEVFOR  before 
OPEVAL? 

VII •  Manpower,  Personnel  and  Training  Support 

a.  If  training  facilities  are  required,  will  they  be  ready  for 
IOC  (initial  operational  capability)? 

b.  Has  procurement  been  planned  for  systems/equipment  for  train 
xng  purposes? 

c.  Does  the  ILS  Plan  provide  for  delivery  of  the  system  to  the 
appropriate  training  site  for  installation  before  initial  training? 

d.  Has  training  been  planned  for  OPEVAL  personnel  and  system 
users  during  the  system's  life  cycle? 

e.  If  a  Navy  Training  Plan  is  required,  will  it  be  available 
for  OPEVAL? 

f.  Does  the  crew  scheduling  and  phasing  plan  allow  sufficient 
personnel  to  be  trained  and  on  board  before  IOC? 


IOC? 


g.  Have  Navy  schools  or  courses  actually  been  established? 

h.  Is  an  up-to-date  table  provided  (as  required  by  OPHAVINST 
4100.3A)  that  summarises  total  manpower  resources  required  to 
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operate  and  maintain  the  system  through  the  first  10  years  of  opera¬ 
tion? 


i.  Are  personnel  and  training  requirements  discussed  for  opera¬ 
tion,  calibration,  and  repair  of  the  various  types  of  test  equip¬ 
ment? 
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